From wflanagan at gmail.com Mon Jan 14 10:25:16 2008 From: wflanagan at gmail.com (William Flanagan) Date: Mon, 14 Jan 2008 10:25:16 -0500 Subject: [Mechanize-users] Quick and Easy Question Message-ID: <321613ee0801140725k58018052m16655ebbb5237e33@mail.gmail.com> All, Been a while since I've "mechanized" and I'm running into an idiot simple problem. I'm looking for more detailed examples to refresh my memory. My question is how to get a list of links off of a page based on information in the img src attribute, specifically if the img tag contains a "mp3.gif/jpg". I, for the life of me, can't seem to remember how to do this. And, as I go through this exercise, I am sure I will run into other issues as well. So, if anyone has any good sources for examples, I'd love to see them. Thanks, William -------------- next part -------------- An HTML attachment was scrubbed... URL: http://rubyforge.org/pipermail/mechanize-users/attachments/20080114/53f77921/attachment.html From aaron at tenderlovemaking.com Mon Jan 14 19:55:05 2008 From: aaron at tenderlovemaking.com (Aaron Patterson) Date: Mon, 14 Jan 2008 16:55:05 -0800 Subject: [Mechanize-users] Quick and Easy Question In-Reply-To: <321613ee0801140725k58018052m16655ebbb5237e33@mail.gmail.com> References: <321613ee0801140725k58018052m16655ebbb5237e33@mail.gmail.com> Message-ID: <20080115005505.GA18437@mac-mini.lan> Hey William! On Mon, Jan 14, 2008 at 10:25:16AM -0500, William Flanagan wrote: > All, > > Been a while since I've "mechanized" and I'm running into an idiot simple > problem. I'm looking for more detailed examples to refresh my memory. > > My question is how to get a list of links off of a page based on information > in the img src attribute, specifically if the img tag contains a > "mp3.gif/jpg". > I, for the life of me, can't seem to remember how to do this. > > And, as I go through this exercise, I am sure I will run into other issues > as well. So, if anyone has any good sources for examples, I'd love to see > them. Give something like this a try: agent = WWW::Mechanize.new page = agent.get('http://localhost/~aaron/test.html') links = page.search('//a').select { |link| img = link.search('//img').first img && img['src'] =~ /mp3\.(gif|jpg)/ } Page can be searched like Hpricot, and links found through that interface can be clicked. So feel free to get specific with your Hpricot queries. -- Aaron Patterson http://tenderlovemaking.com/ From aaron at tenderlovemaking.com Tue Jan 15 00:18:20 2008 From: aaron at tenderlovemaking.com (Aaron Patterson) Date: Mon, 14 Jan 2008 21:18:20 -0800 Subject: [Mechanize-users] [ANN] mechanize 0.7.0 Released Message-ID: <20080115051820.GA20773@mac-mini.lan> mechanize version 0.7.0 has been released! * The Mechanize library is used for automating interaction with websites. Mechanize automatically stores and sends cookies, follows redirects, can follow links, and submit forms. Form fields can be populated and submitted. Mechanize also keeps track of the sites that you have visited as a history. Changes: # Mechanize CHANGELOG ## 0.7.0 * Removed Ruby 1.8.2 support * Changed parser to lazily parse links * Lazily parsing document * Adding verify_callback for SSL requests. Thanks Mike Dalessio! * Fixed a bug with Accept-Language header. Thanks Bill Siggelkow. * -- Aaron Patterson http://tenderlovemaking.com/ From lrlebron at gmail.com Tue Jan 15 11:14:24 2008 From: lrlebron at gmail.com (Luis Lebron) Date: Tue, 15 Jan 2008 10:14:24 -0600 Subject: [Mechanize-users] Converting Watir script to Mechanize Message-ID: <2d0d0c9f0801150814s623519cdx38849c0396262438@mail.gmail.com> I have a Watir script that I would like to convert to a Mechanize script. The watir code looks like this require 'watir' include Watir def test ie = IE.new ie.goto('http://cpref.gsm.com/inter.asp?r=8084') ie.text_field(:name, 'inter_string').set('Potassium') ie.button(:id, 'image1').click ie.link(:text, /Potassium/).click ie.image(:src, 'http://cpref.gsm.com/images/plus_yellow.gif').click ie.text_field(:name, 'inter_string').set('Xopenex') ie.button(:id, 'image1').click ie.link(:text, /Xopenex/).click ie.image(:src, 'http://cpref.gsm.com/images/plus_yellow.gif').click ie.text_field(:name, 'inter_string').set('Lasix') ie.button(:id, 'image1').click ie.link(:text, /Lasix/).click ie.link(:url, 'javascript:document.selectForm.submit()').click end I started working on the Mechanize script but it is throw the following error: c:/ruby/lib/ruby/gems/1.8/gems/mechanize-0.6.11/lib/mechanize/form.rb:302:in `method_missing': undefined method `inter_string=' for # (NoMethodError) Here the script I'm testing require 'mechanize' agent = WWW::Mechanize.new page = agent.get('http://cpref.gsm.com/inter.asp?r=8084') search_form = page.forms.with.name("interdrugs").first search_form.inter_string = "Potassium" search_form.submit puts page.body Any help would be greatly appreciated. Luis From aaron at tenderlovemaking.com Tue Jan 15 16:39:56 2008 From: aaron at tenderlovemaking.com (Aaron Patterson) Date: Tue, 15 Jan 2008 13:39:56 -0800 Subject: [Mechanize-users] Converting Watir script to Mechanize In-Reply-To: <2d0d0c9f0801150814s623519cdx38849c0396262438@mail.gmail.com> References: <2d0d0c9f0801150814s623519cdx38849c0396262438@mail.gmail.com> Message-ID: <20080115213956.GA29053@mac-mini.lan> Hi Luis, On Tue, Jan 15, 2008 at 10:14:24AM -0600, Luis Lebron wrote: > I have a Watir script that I would like to convert to a Mechanize > script. The watir code looks like this > > require 'watir' > include Watir > > def test > ie = IE.new > ie.goto('http://cpref.gsm.com/inter.asp?r=8084') > > ie.text_field(:name, 'inter_string').set('Potassium') > ie.button(:id, 'image1').click > ie.link(:text, /Potassium/).click > ie.image(:src, 'http://cpref.gsm.com/images/plus_yellow.gif').click > > ie.text_field(:name, 'inter_string').set('Xopenex') > ie.button(:id, 'image1').click > ie.link(:text, /Xopenex/).click > ie.image(:src, 'http://cpref.gsm.com/images/plus_yellow.gif').click > > ie.text_field(:name, 'inter_string').set('Lasix') > ie.button(:id, 'image1').click > ie.link(:text, /Lasix/).click > > ie.link(:url, 'javascript:document.selectForm.submit()').click > end > > > I started working on the Mechanize script but it is throw the following error: > c:/ruby/lib/ruby/gems/1.8/gems/mechanize-0.6.11/lib/mechanize/form.rb:302:in > `method_missing': undefined method `inter_string=' for > # (NoMethodError) > > Here the script I'm testing > > require 'mechanize' > agent = WWW::Mechanize.new > page = agent.get('http://cpref.gsm.com/inter.asp?r=8084') > search_form = page.forms.with.name("interdrugs").first > search_form.inter_string = "Potassium" > search_form.submit > puts page.body > > Any help would be greatly appreciated. The problem is most likely a bug in hpricot. Hpricot has problems parsing forms in tables, so those methods may not get added to the form object. If you know that that form field is supposed to be on the form object, just treat the form object like a hash, and it will add the field if its missing. For example: search_form['inter_string'] = "Potassium" Hope that helps. -- Aaron Patterson http://tenderlovemaking.com/ From mensa200 at gmx.net Mon Jan 14 16:38:47 2008 From: mensa200 at gmx.net (mensa200) Date: Mon, 14 Jan 2008 22:38:47 +0100 Subject: [Mechanize-users] Problems with forms Message-ID: <20080114223847.zm286rug6os0kgkc@obelix.kohlhaupt.ch> hello all i'm a newbie on mechanize, but already had some success. But now i'm in a real problem : The following HTML-Code is not parsed corect, not all form element are found. I'can't change the server-code. How to fix ? HTML-Site ------X-----------------------------------------------------------------------X- ricardo.ch - Artikel verkaufen
ricardo.chAndere Artikel verkaufen
Artikel verkaufen
M?chten Sie einen Artikel verkaufen? Dann f?llen Sie dieses Formular aus. Alle mit * gekennzeichneten Felder sind obligatorisch.

Bitte beachten Sie, dass Sie zuerst den Aktivierungscode eingeben m?ssen, um etwas verkaufen zu k?nnen. ?berpr?fen Sie zudem vor dem Verkauf in den AGB die laufend aktualisierte Liste der verbotenen Produkte.

Wenn Sie ein Auto oder ein Motorrad verkaufen m?chten, klicken Sie auf das entsprechende Feld.

Andere Artikel verkaufenEin Auto verkaufenEin Motorrad verkaufen
Kategorie-Auswahl

Hauptkategorie:  *    

 Hilfe bei Kategorie-Auswahl
W?hlen Sie die Kategorien von links nach rechts f?r die Platzierung Ihres Artikels. Auswahl ?ndern

 
ProduktbeschreibungHilfe
Name der VorlageMit einer Vorlage k?nnen Sie Ihr Angebot optimaler pr?sentieren (kostenlos).
Produkttitel *
Geben Sie einen Titel ein mit maximal 60 Zeichen, der alle wichtigen W?rter enth?lt.
Beschreibung *

 



Hinweis: Es ist nicht erlaubt im Titel, Text und in Bildern E-Mail-Adressen oder www-Adressen zu integrieren. Beachten Sie unsere Liste der verbotenen Produkte.

Im Feld der vergr?sserten Ansicht k?nnen Sie Ihre Texte optimal gestalten (direkte Eingabe) oder bereits formatierte Texte einf?gen.

Beschreiben Sie Ihren Artikel so genau und umfassend wie m?glich. Erw?hnen Sie wichtige Details wie Besch?digungen o.?.
  Link einf?gen zu allen Ihren offenen Angeboten    
GarantieGarantieleistung f?r dieses Angebot
Anzahl *Achtung! Anzahl Artikel f?r dieses Angebot
Zustand *Zustand des Artikels
Ihre ReferenzMit Ihrer Referenz k?nnen Sie Ihre Artikel intern kennzeichnen (= interne Referenznummer). Die Referenz kann maximal 20 Zeichen enthalten.
Aktuelle Liste ?
Preis pro Bild
 Maximal 500 KB pro Bild
 
Angebotsart, Preise und LieferkonditionenHilfe
Angebotsart
Auktion (max. 10 Tage)
Fixpreis-Angebot (max. 10 Tage)
Einen Artikel versteigern oder zu einem festen Preis anbieten
  
Mindestpreis *
CHF . 
Erh?hungsschritt *
CHF . 
  
Verf?gbarkeit
  Ab Lager / sofort verf?gbar
  

Artikel-Verf?gbarkeit
Lieferkonditionen CH *Geben Sie an, wie der Artikel geliefert wird
Lieferkosten CH:. Wenn die Transportkosten zu Lasten des K?ufers gehen, m?ssen diese Kosten zwingend angegeben werden.
 
ZahlungskonditionenHilfe
Zahlungskondition *Geben Sie an, wie die Zahlung erfolgen soll
  
Akzeptierte Zahlungsweisen Bezahlung per E-Mail - Moneybookers

Treuhandservice - moneybookers.com

Postzahlung

Kreditkarte
Zus?tzliche Zahlungsweisen, die Sie akzeptieren
 
Erh?hen Sie Ihren Verkaufserfolg durch den Einsatz von geb?hrenpflichtigen Einstelloptionen!Hilfe
Bildergalerie
 
(Ihr Angebot erscheint mit Bild in den Artikellisten)  
Anzeige eines Beispiels
Fetter Titel
 
(Ihr Produkttitel erscheint mit fetter Schrift)
Anzeige eines Beispiels
Farbiger Hintergrund
 
(Ihr Angebot erscheint mit farbigem Hintergrund)
Anzeige eines Beispiels
Top-Angebot Hauptkategorie
 
(Ihr Angebot erscheint auf der Hauptkategorienseite)
Anzeige eines Beispiels
Top-Angebot Startseite
 
(Ihr Angebot erscheint auf der Startseite und auf der Hauptkategorienseite)
Anzeige eines Beispiels
 
Startzeit, Angebotsdauer und ReaktivierungHilfe
Startzeit * Angebot sofort einstellenW?hlen Sie, wann Ihr Angebot beginnen soll
Angebot starten am:
 
  
Angebotsdauer *
 Tag(e)  Stunde(n)
Mindestdauer: 1 Tag
  
Automatische Reaktivierung
Wenn das Angebot beim ersten Mal nicht erfolgreich endet,
das Angebot -mal reaktivieren.
 
Erinnerung
Erinnerung per E-Mail Minuten vor Angebotsende
 
AGB *
Ich habe die Allgemeinen Gesch?ftsbedingungen (AGB) gelesen und erkl?re mich damit einverstanden.
  
Eingabe speichern
Dieselben Verkaufskonditionen f?r n?chste Angebote einsetzen.
 
 
------------X-------------------------------------------X---------------------- what mechanize or hpricot is seeing the sellingForm (the problem) ------------X-----------------------------------------X------------------ # # # # # # # # # # # # # # # # # # # # # # # # # # #, @selected=false, @text="W\344hlen Sie...", @value="">, #, @selected=false, @text="Antiquit\344ten & Kunst", @value="38399">, #, @selected=false, @text="Auto & Motorrad", @value="38567">, #, @selected=false, @text="Audio, TV & Video", @value="63787">, #, @selected=false, @text="Briefmarken", @value="38766">, #, @selected=false, @text="B\374cher & Comics", @value="38889">, #, @selected=false, @text="B\374ro & Gewerbe", @value="63788">, #, @selected=false, @text="Computer & Netzwerk", @value="39091">, #, @selected=false, @text="Filme & DVD", @value="39349">, #, @selected=false, @text="Foto & Optik", @value="39460">, #, @selected=false, @text="Games & Spielkonsolen", @value="39563">, #, @selected=false, @text="Handwerk & Garten", @value="39825">, #, @selected=false, @text="Handy, Festnetz, Funk", @value="39940">, #, @selected=false, @text="Haushalt & Wohnen", @value="40295">, #, @selected=false, @text="Kind & Baby", @value="40520">, #, @selected=false, @text="Kleidung & Accessoires", @value="40748">, #, @selected=false, @text="Kosmetik & Pflege", @value="41057">, #, @selected=false, @text="Modellbau & Hobby", @value="41126">, #, @selected=false, @text="M\374nzen", @value="41260">, #, @selected=false, @text="Musik & Musikinstrumente", @value="63781">, #, @selected=false, @text="Sammeln & Seltenes", @value="41518">, #, @selected=false, @text="Spielzeug & Basteln", @value="41733">, #, @selected=false, @text="Sports", @value="41875">, #, @selected=false, @text="Tickets & Gutscheine", @value="42099">, #, @selected=false, @text="Tierzubeh\366r", @value="42135">, #, @selected=false, @text="Uhren & Schmuck", @value="42272">, #, @selected=false, @text="Wein & Genuss", @value="42454">], @value=[]> #, @selected=false, @text="--------------------------------------------------", @value=nil>], @value=[]> #, @selected=false, @text="--------------------------------------------------", @value=nil>], @value=[]> #, @selected=false, @text="--------------------------------------------------", @value=nil>], @value=[]> #, @selected=false, @text="--------------------------------------------------", @value=nil>], @value=[]> #, @selected=true, @text="Kein ", @value="-1">], @value=[]> #, @selected=false, @text="Bitte w\344hlen Sie...", @value="-1">, #, @selected=false, @text="Gem\344ss Beschreibung", @value="0">, #, @selected=true, @text="Keine Garantie", @value="1">], @value=[]> #, @selected=false, @text="Bitte w\344hlen Sie...", @value="-1">, #, @selected=false, @text="Fabrikneu", @value="0">, #, @selected=false, @text="Fabrikneu und unge\366ffnet", @value="1">, #, @selected=true, @text="Neuwertig", @value="2">, #, @selected=false, @text="Gebraucht", @value="3">, #, @selected=false, @text="Antik", @value="4">, #, @selected=false, @text="Defekt", @value="5">], @value=[]> #, @selected=false, @text="1-5 Arbeitstage", @value="0">, #, @selected=false, @text="6-10 Arbeitstage", @value="1">, #, @selected=false, @text="11-14 Arbeitstage", @value="2">, #, @selected=false, @text="15-20 Arbeitstage", @value="3">, #, @selected=false, @text="2 Monaten", @value="4">], @value=[]> #, @selected=false, @text="Bitte w\344hlen Sie...", @value="-1">, #, @selected=false, @text="Gem\344ss Beschreibung", @value="0">, #, @selected=true, @text="Brief A-Post", @value="1">, #, @selected=false, @text="Brief B-Post", @value="2">, #, @selected=false, @text="Paket A-Post", @value="3">, #, @selected=false, @text="Paket B-Post", @value="4">, #, @selected=false, @text="Einschreiben", @value="5">, #, @selected=false, @text="Nachnahme", @value="6">, #, @selected=false, @text="Spediteur oder Kurier", @value="7">, #, @selected=false, @text="Abholung durch K\344ufer", @value="8">, #, @selected=false, @text="Lieferung durch Anbieter", @value="9">], @value=[]> #, @selected=false, @text="Bitte w\344hlen Sie...", @value="-1">, #, @selected=false, @text="Gem\344ss Beschreibung", @value="0">, #, @selected=false, @text="Barzahlung", @value="1">, #, @selected=false, @text="Rechnung nach Erhalt zahlbar", @value="2">, #, @selected=false, @text="Rechnung innert 14 Tagen zahlbar", @value="3">, #, @selected=false, @text="Rechnung innert 30\240 Tagen zahlbar", @value="4">, #, @selected=true, @text="Vorauszahlung", @value="5">], @value=[]>} {radiobuttons} {checkboxes # # # # # # # # # # # # # # # #} {file_uploads} {buttons}> --------------------X--------------------------------X-------------------- any idea ? Thanks Mrks From ngrandy at gmail.com Wed Jan 16 04:22:26 2008 From: ngrandy at gmail.com (Nick Grandy) Date: Wed, 16 Jan 2008 01:22:26 -0800 Subject: [Mechanize-users] settings headers in mechanize Message-ID: <9a6cef0c0801160122o13d17e1fobbd644396d874d12@mail.gmail.com> hi, a potential mechanize user here. i've looked at the API but it's not clear what capacity there is to arbitrarily set the headers - does mechanize allow for this? mechanize is clearly great for web browsing, but i also need to construct get and post requests from scratch. i've looked at net:http, libcurl, openuri, but mechanize seems simpler and more functional than the others - is this an appropriate thing to do with mechanize? any tips? thanks, nick From mortee.lists at kavemalna.hu Wed Jan 16 12:57:36 2008 From: mortee.lists at kavemalna.hu (mortee) Date: Wed, 16 Jan 2008 18:57:36 +0100 Subject: [Mechanize-users] settings headers in mechanize In-Reply-To: <9a6cef0c0801160122o13d17e1fobbd644396d874d12@mail.gmail.com> References: <9a6cef0c0801160122o13d17e1fobbd644396d874d12@mail.gmail.com> Message-ID: I'm mostly just guessing here, because I haven't gone to great lengths to explore the possibilities. But it seems like you won't easily set arbitrary request headers, because it's generated and used whithin the mechanize methods, and you don't have direct access to it. >From version 0.6.10: # Fetches the URL passed in and returns a page. def get(url, referer=nil, &block) cur_page = referer || current_page || Page.new( nil, {'content-type'=>'text/html'}) # fetch the page abs_uri = to_absolute_uri(url, cur_page) request = fetch_request(abs_uri) page = fetch_page(abs_uri, request, cur_page, &block) add_to_history(page) page end As you can see, the request is generated and used immediately here - and the fetch_request and fetch_page methods are private, you can't directly use them. If I really wanted to use mechanize AND directly manipulate the headers then I'd call those private methods using instance_eval, which allows to do that - but since those are not part of the public interface, they are more likely to change with a new version, and break your code. mortee Nick Grandy wrote: > hi, > a potential mechanize user here. i've looked at the API but it's not > clear what capacity there is to arbitrarily set the headers - does > mechanize allow for this? > mechanize is clearly great for web browsing, but i also need to > construct get and post requests from scratch. i've looked at > net:http, libcurl, openuri, but mechanize seems simpler and more > functional than the others - is this an appropriate thing to do with > mechanize? any tips? > thanks, > nick From aaron at tenderlovemaking.com Thu Jan 17 19:05:45 2008 From: aaron at tenderlovemaking.com (Aaron Patterson) Date: Thu, 17 Jan 2008 16:05:45 -0800 Subject: [Mechanize-users] settings headers in mechanize In-Reply-To: <9a6cef0c0801160122o13d17e1fobbd644396d874d12@mail.gmail.com> References: <9a6cef0c0801160122o13d17e1fobbd644396d874d12@mail.gmail.com> Message-ID: <20080118000545.GA21600@mac-mini.lan> On Wed, Jan 16, 2008 at 01:22:26AM -0800, Nick Grandy wrote: > hi, > a potential mechanize user here. i've looked at the API but it's not > clear what capacity there is to arbitrarily set the headers - does > mechanize allow for this? > mechanize is clearly great for web browsing, but i also need to > construct get and post requests from scratch. i've looked at > net:http, libcurl, openuri, but mechanize seems simpler and more > functional than the others - is this an appropriate thing to do with > mechanize? any tips? Its not very easy to set custom headers right now. I'm definitly open to API suggestions on this one. Basically, the best way right now is to subclass Mechanize and implement the set_headers method. For example: class MyMech < WWW::Mechanize def set_headers(uri, request, cur_page) super request.add_field('Foo', 'bar') end end agent = MyMech.new page = ...... -- Aaron Patterson http://tenderlovemaking.com/ From aaron at tenderlovemaking.com Thu Jan 17 19:07:25 2008 From: aaron at tenderlovemaking.com (Aaron Patterson) Date: Thu, 17 Jan 2008 16:07:25 -0800 Subject: [Mechanize-users] Problems with forms In-Reply-To: <20080114223847.zm286rug6os0kgkc@obelix.kohlhaupt.ch> References: <20080114223847.zm286rug6os0kgkc@obelix.kohlhaupt.ch> Message-ID: <20080118000725.GB21600@mac-mini.lan> On Mon, Jan 14, 2008 at 10:38:47PM +0100, mensa200 wrote: > hello all > > i'm a newbie on mechanize, but already had some success. > > But now i'm in a real problem : > > The following HTML-Code is not parsed corect, not all form element are > found. I'can't change the server-code. > > How to fix ? If you know what the field names are, just treat the form object as a hash. For example: form = page.forms.first form['missing_field'] = 'asdfasdf' -- Aaron Patterson http://tenderlovemaking.com/ From ngrandy at gmail.com Fri Jan 18 12:43:22 2008 From: ngrandy at gmail.com (Nick Grandy) Date: Fri, 18 Jan 2008 09:43:22 -0800 Subject: [Mechanize-users] settings headers in mechanize In-Reply-To: <20080118000545.GA21600@mac-mini.lan> References: <9a6cef0c0801160122o13d17e1fobbd644396d874d12@mail.gmail.com> <20080118000545.GA21600@mac-mini.lan> Message-ID: <9a6cef0c0801180943k4c673087sc81e4a94a288219c@mail.gmail.com> Aaron and mortee, Thanks for the tips. Looks like this is a reasonable hack to make, especially to make use of the other functionality of mechanize. Nick On Jan 17, 2008 4:05 PM, Aaron Patterson wrote: > > On Wed, Jan 16, 2008 at 01:22:26AM -0800, Nick Grandy wrote: > > hi, > > a potential mechanize user here. i've looked at the API but it's not > > clear what capacity there is to arbitrarily set the headers - does > > mechanize allow for this? > > mechanize is clearly great for web browsing, but i also need to > > construct get and post requests from scratch. i've looked at > > net:http, libcurl, openuri, but mechanize seems simpler and more > > functional than the others - is this an appropriate thing to do with > > mechanize? any tips? > > Its not very easy to set custom headers right now. I'm definitly open > to API suggestions on this one. Basically, the best way right now is to > subclass Mechanize and implement the set_headers method. > > For example: > > class MyMech < WWW::Mechanize > def set_headers(uri, request, cur_page) > super > request.add_field('Foo', 'bar') > end > end > > agent = MyMech.new > page = ...... > > -- > Aaron Patterson > http://tenderlovemaking.com/ > > _______________________________________________ > Mechanize-users mailing list > Mechanize-users at rubyforge.org > http://rubyforge.org/mailman/listinfo/mechanize-users > -- Nick Grandy mobile: (+1) 347-835-1706