From lee.hambley at gmail.com Thu Jul 2 12:04:36 2009 From: lee.hambley at gmail.com (Lee Hambley) Date: Thu, 2 Jul 2009 17:04:36 +0100 Subject: [Mechanize-users] "save" method In-Reply-To: References: Message-ID: Khomich, Use Net::HTTP. - Lee 2009/7/2 Khomich Vlad > Hi guys. i`m using mechanize to save the images ( i have problems using > open-uri ) , with > a=agent.get(link) > a.save > > do you know how can i provide a filename to save as? Mechanize sets the > default name > > _______________________________________________ > Mechanize-users mailing list > Mechanize-users at rubyforge.org > http://rubyforge.org/mailman/listinfo/mechanize-users > -------------- next part -------------- An HTML attachment was scrubbed... URL: From astarr at wiredquote.com Sun Jul 5 22:35:13 2009 From: astarr at wiredquote.com (Aaron Starr) Date: Sun, 5 Jul 2009 19:35:13 -0700 Subject: [Mechanize-users] Encoding in meta http-equiv tags, but not in response headers Message-ID: <669cc1ca0907051935r53bb1945qe1f6a32ce97a188a@mail.gmail.com> Hey, all, I just ran into a situation where a web site is not specifying a character encoding in the response headers. It then specifies UTF-8 in a content-type meta tag. Mechanize doesn't read the meta tags to find the encoding, and ends up using an encoding that doesn't work, ISO-8859-1. Here's a quick monkey patch that checks for a meta http-equiv tag in the page body and uses it to set the encoding if the encoding isn't specified in the response headers: http://pastie.org/535231 I don't have time to pretend to be smart by making this pretty, but I think it may still be useful for someone, and similar functionality would be welcome in the core sources. Aaron -------------- next part -------------- An HTML attachment was scrubbed... URL: From aaron.patterson at gmail.com Sun Jul 5 23:17:58 2009 From: aaron.patterson at gmail.com (Aaron Patterson) Date: Sun, 5 Jul 2009 20:17:58 -0700 Subject: [Mechanize-users] Encoding in meta http-equiv tags, but not in response headers In-Reply-To: <669cc1ca0907051935r53bb1945qe1f6a32ce97a188a@mail.gmail.com> References: <669cc1ca0907051935r53bb1945qe1f6a32ce97a188a@mail.gmail.com> Message-ID: <6959e1680907052017m65321840k988c2c95eded4723@mail.gmail.com> On Sun, Jul 5, 2009 at 7:35 PM, Aaron Starr wrote: > Hey, all, > I just ran into a situation where a web site is not specifying a character > encoding in the response headers. It then specifies UTF-8 in a content-type > meta tag. Mechanize doesn't read the meta tags to find the encoding, and > ends up using an encoding that doesn't work, ISO-8859-1. > Here's a quick monkey patch that checks for a meta http-equiv tag in the > page body and uses it to set the encoding if the encoding isn't specified in > the response headers: > ?? ?http://pastie.org/535231 > I don't have time to pretend to be smart by making this pretty, but I think > it may still be useful for someone, and similar functionality would be > welcome in the core sources. Can you write a test to reproduce this error? Nokogiri *should* pick up the encoding from the meta tag. Can you try printing out: puts page.parser.encoding That should contain the encoding that nokogiri intuited. -- Aaron Patterson http://tenderlovemaking.com/ From astarr at wiredquote.com Sun Jul 5 23:55:33 2009 From: astarr at wiredquote.com (Aaron Starr) Date: Sun, 5 Jul 2009 20:55:33 -0700 Subject: [Mechanize-users] Encoding in meta http-equiv tags, but not in response headers In-Reply-To: <6959e1680907052017m65321840k988c2c95eded4723@mail.gmail.com> References: <669cc1ca0907051935r53bb1945qe1f6a32ce97a188a@mail.gmail.com> <6959e1680907052017m65321840k988c2c95eded4723@mail.gmail.com> Message-ID: <669cc1ca0907052055m26cedfai68d98934c627714a@mail.gmail.com> > > > I don't have time to pretend to be smart by making this pretty [...] > > Can you write a test to reproduce this error? Nokogiri *should* pick > up the encoding from the meta tag. > > Can you try printing out: > > puts page.parser.encoding > > That should contain the encoding that nokogiri intuited. Sure, when I get a chance I'll fuss with it some more. There is a distinct difference in behavior for me, though, with and without the monkey patch. Unfortunately, I can't just direct you to the page, because it's hidden behind passwords and byzantine AJAX protocols, and whatnot. Aaron -------------- next part -------------- An HTML attachment was scrubbed... URL: From jhuizingh at mindless.com Fri Jul 10 19:33:49 2009 From: jhuizingh at mindless.com (Jonathan Huizingh) Date: Fri, 10 Jul 2009 18:33:49 -0500 Subject: [Mechanize-users] Iframes in pages Message-ID: Does mechanize handle iframes that show up within pages that it loads? I have a page that loads a form I need to fill out within an iframe. Is it possible to do something like this? : outer_page = agent.get('http://path.to.site.come/outer_page.html') form = outer_page.iframes[0].forms[0] .... more code ... Is there some way to do something like that? If not, is there a recommended way to handle forms embedded in iframes? Thanks, Jonathan Huizingh Pronunciation: HI-zing From wojciech at piekutowski.net Sun Jul 12 05:51:52 2009 From: wojciech at piekutowski.net (Wojciech Piekutowski) Date: Sun, 12 Jul 2009 11:51:52 +0200 Subject: [Mechanize-users] Mechanize drop down menus In-Reply-To: References: Message-ID: On Sun, Jul 12, 2009 at 01:33, Ferlander Davids wrote: > I'm having trouble with a drop down menu. I can't find much mentioned in online references about drop down menus. > I've tried a bunch of different things, right now it's > form = page.form("term_in").first > > but that's returning?nil. You need to find out what are the elements that render the menu. The best approach is to look at the source code of the web page in question and search for the menu part. I don't think that your menu is stored in
, more probable are some
    or
    tags. Firebug is often useful for this kind of tasks. Greetings, Wojciech Piekutowski From wojciech at piekutowski.net Sun Jul 12 06:44:12 2009 From: wojciech at piekutowski.net (Wojciech Piekutowski) Date: Sun, 12 Jul 2009 12:44:12 +0200 Subject: [Mechanize-users] Mechanize drop down menus In-Reply-To: References: Message-ID: On Sun, Jul 12, 2009 at 12:20, Ferlander Davids wrote: > I've looked a little more carefully now. It looks like it's just - field = form.field_with(:name => 'term_in') - tick the option you want to go to - field.options[n].select - submit the form - form.submit Please take a look at the docs: http://mechanize.rubyforge.org/mechanize/EXAMPLES_rdoc.html http://mechanize.rubyforge.org/mechanize/GUIDE_rdoc.html Greetings, Wojciech Piekutowski > Thanks for your reply! > Elliot > > > > Select a Term: > > > > > > > >   > > > > > > > > > > On Sun, Jul 12, 2009 at 2:51 AM, Wojciech Piekutowski > wrote: >> >> On Sun, Jul 12, 2009 at 01:33, Ferlander Davids >> wrote: >> > I'm having trouble with a drop down menu. I can't find much mentioned in >> > online references about drop down menus. >> > I've tried a bunch of different things, right now it's >> > form = page.form("term_in").first >> > >> > but that's returning?nil. >> >> You need to find out what are the elements that render the menu. The >> best approach is to look at the source code of the web page in >> question and search for the menu part. I don't think that your menu is >> stored in
    , more probable are some
      or
      tags. Firebug >> is often useful for this kind of tasks. >> >> Greetings, >> Wojciech Piekutowski >> _______________________________________________ >> Mechanize-users mailing list >> Mechanize-users at rubyforge.org >> http://rubyforge.org/mailman/listinfo/mechanize-users > > _______________________________________________ > Mechanize-users mailing list > Mechanize-users at rubyforge.org > http://rubyforge.org/mailman/listinfo/mechanize-users > From astarr at wiredquote.com Thu Jul 16 15:12:46 2009 From: astarr at wiredquote.com (Aaron Starr) Date: Thu, 16 Jul 2009 12:12:46 -0700 Subject: [Mechanize-users] Use mechanize to POST arbitrary data? In-Reply-To: References: Message-ID: <669cc1ca0907161212y49792507w90397d7bae50ad24@mail.gmail.com> John, I did this in a hacky little script: class JsonForm ??def initialize(json) ?? ?@json = json ??end ??def page ?? ?nil ??end ??def request_data ?? ?@json ??end ??def enctype ?? ?'application/json' ??end end class WWW::Mechanize ??def post_json(url, json) ?? ?post_form(url, JsonForm.new(json)) ??end end Then: web_agent.post_json('/blah/blah', '{"some":"json"}') If there's a better way, I'd be interested, too. Aaron On Thu, Jul 16, 2009 at 11:50 AM, John Norman wrote: > > I wanted to log this in the feature tracker at rubyforge, but rubyforge won't accept my password change. :-( In any case: > > I would like to do a post, but have the body of the post be some arbitrary text (in fact, I'm posting escaped JSON). > > E.g., > > I want to do something like: > > agent.post_raw('/ajaxprocessor', stuff) > > where stuff is a string with value: > > %7B%22promocode%22%3A%20%22129182%22%2C%20%22zipcode%22%3A%20%22%22%2C%20%22iscobrandselected%22%3A%20false%2C%20%22shippingoptions%22%3A%20%7B%224164011%22%3A%20%22GROUND4164011%22%7D%7D > > To be sure, I could do this with Net::HTTP, but I want to leverage my existing state in Mechanize (cookies, etc.). > > This doesn't look like an easy hack in Mechanize, because it would expect fetch_page to do something different with the params (i.e., no params). > > Any ideas? > > John > > > _______________________________________________ > Mechanize-users mailing list > Mechanize-users at rubyforge.org > http://rubyforge.org/mailman/listinfo/mechanize-users From astarr at wiredquote.com Thu Jul 16 15:49:45 2009 From: astarr at wiredquote.com (Aaron Starr) Date: Thu, 16 Jul 2009 12:49:45 -0700 Subject: [Mechanize-users] Use mechanize to POST arbitrary data? In-Reply-To: References: <669cc1ca0907161212y49792507w90397d7bae50ad24@mail.gmail.com> Message-ID: <669cc1ca0907161249i615b7339q6c8c5524c9c7c5ff@mail.gmail.com> John, I'm sure Aaron Patterson would have a more elegant solution -- he probably won't want to be blamed for this one. Some Other Aaron On Thu, Jul 16, 2009 at 12:17 PM, John Norman wrote: > Aaron, > > As always, THANKS. > > John > > > On Thu, Jul 16, 2009 at 3:12 PM, Aaron Starr wrote: > >> John, >> >> I did this in a hacky little script: >> >> class JsonForm >> def initialize(json) >> @json = json >> end >> def page >> nil >> end >> def request_data >> @json >> end >> def enctype >> 'application/json' >> end >> end >> class WWW::Mechanize >> def post_json(url, json) >> post_form(url, JsonForm.new(json)) >> end >> end >> >> Then: web_agent.post_json('/blah/blah', '{"some":"json"}') >> >> If there's a better way, I'd be interested, too. >> >> Aaron >> >> >> >> On Thu, Jul 16, 2009 at 11:50 AM, John Norman wrote: >> > >> > I wanted to log this in the feature tracker at rubyforge, but rubyforge >> won't accept my password change. :-( In any case: >> > >> > I would like to do a post, but have the body of the post be some >> arbitrary text (in fact, I'm posting escaped JSON). >> > >> > E.g., >> > >> > I want to do something like: >> > >> > agent.post_raw('/ajaxprocessor', stuff) >> > >> > where stuff is a string with value: >> > >> > >> %7B%22promocode%22%3A%20%22129182%22%2C%20%22zipcode%22%3A%20%22%22%2C%20%22iscobrandselected%22%3A%20false%2C%20%22shippingoptions%22%3A%20%7B%224164011%22%3A%20%22GROUND4164011%22%7D%7D >> > >> > To be sure, I could do this with Net::HTTP, but I want to leverage my >> existing state in Mechanize (cookies, etc.). >> > >> > This doesn't look like an easy hack in Mechanize, because it would >> expect fetch_page to do something different with the params (i.e., no >> params). >> > >> > Any ideas? >> > >> > John >> > >> > >> > _______________________________________________ >> > Mechanize-users mailing list >> > Mechanize-users at rubyforge.org >> > http://rubyforge.org/mailman/listinfo/mechanize-users >> _______________________________________________ >> Mechanize-users mailing list >> Mechanize-users at rubyforge.org >> http://rubyforge.org/mailman/listinfo/mechanize-users > > > > _______________________________________________ > Mechanize-users mailing list > Mechanize-users at rubyforge.org > http://rubyforge.org/mailman/listinfo/mechanize-users > -------------- next part -------------- An HTML attachment was scrubbed... URL: From lee.hambley at gmail.com Fri Jul 24 05:21:23 2009 From: lee.hambley at gmail.com (Lee Hambley) Date: Fri, 24 Jul 2009 10:21:23 +0100 Subject: [Mechanize-users] Processing dialogue boxes In-Reply-To: <4A69260C.4060104@roscommonhq.com> References: <4A69260C.4060104@roscommonhq.com> Message-ID: Ross, I'm afraid I have nothing to offer, after a similar problem we compromised, this is what people mean about making accessible websites. -- Lee Hambley Twitter: @leehambley Blog: http://lee.hambley.name/ Working with Rails: http://is.gd/1s5W1 2009/7/24 Ross Cameron > Have come across a situation where the site I am targeting with Mechanize > responds with a modeless dialogue box which requires an answer by way of a > click to "OK" button. Nothing happens unless you do it would seem. > Are dialog boxes outside the scope of Mechanize, as I can't see any > documentation on them. If not, would appreciate any pointers on how to > process them with Mechanize. > If they are outside of the scope of Mechanize, can anyone offer any > suggestions as to how to deal with them. > > Regards > Ross > > > _______________________________________________ > Mechanize-users mailing list > Mechanize-users at rubyforge.org > http://rubyforge.org/mailman/listinfo/mechanize-users > -------------- next part -------------- An HTML attachment was scrubbed... URL: From wonderdog at gmail.com Fri Jul 24 12:26:22 2009 From: wonderdog at gmail.com (Wonderdog) Date: Fri, 24 Jul 2009 09:26:22 -0700 Subject: [Mechanize-users] ignore ssl verification failures? Message-ID: I need mechanize to ignore the name verification failure of an SSL cert. In this case, I'm using mech to test several servers that are behind a load balancer, but I'm bypassing the LB. Rewriting my hosts file every time I check 1 of the servers is not optimal. :-/ Jon From aaron.patterson at gmail.com Sun Jul 26 00:22:30 2009 From: aaron.patterson at gmail.com (Aaron Patterson) Date: Sat, 25 Jul 2009 21:22:30 -0700 Subject: [Mechanize-users] Failed to build gem native extension In-Reply-To: References: Message-ID: <6959e1680907252122h19f7c0f0y2a5f47ab9f911bbd@mail.gmail.com> On Sat, Jul 25, 2009 at 9:14 PM, Jeffrey Roberts wrote: > Hello all, I have looked up and down on google for a solution to this going > on several days now, I am really hoping someone here can help me out. > > I have all my deps in order, and I believe the error is that it is looking > in /usr/lib when it should be looking in /usr/lib64, I am also unable to > locate the mkmf.log for whatever reason, any ideas on how I can get this > installed? What version of ruby do you have installed? -- Aaron Patterson http://tenderlovemaking.com/ From aaron.patterson at gmail.com Sun Jul 26 01:28:03 2009 From: aaron.patterson at gmail.com (Aaron Patterson) Date: Sat, 25 Jul 2009 22:28:03 -0700 Subject: [Mechanize-users] Failed to build gem native extension In-Reply-To: References: <6959e1680907252122h19f7c0f0y2a5f47ab9f911bbd@mail.gmail.com> Message-ID: <6959e1680907252228m5c365d33p841fcfd4a43c2032@mail.gmail.com> On Sat, Jul 25, 2009 at 9:59 PM, Jeffrey Roberts wrote: > Thank you for such a quick reply, I have 1.8.5, and im noticing its i386, > hrmm, any ideas ? Yes, the current release of nokogiri isn't installing correctly with 1.8.5. You can either upgrade ruby (highly recommend since 1.8.5 isn't supported by the core team anymore), or use an older version of nokogiri like this: $ gem install nokogiri -v 1.3.1 Then gem install mechanize. Newer versions of nokogiri should not suffer from this problem. :-) -- Aaron Patterson http://tenderlovemaking.com/ From aaron.patterson at gmail.com Sun Jul 26 02:32:18 2009 From: aaron.patterson at gmail.com (Aaron Patterson) Date: Sat, 25 Jul 2009 23:32:18 -0700 Subject: [Mechanize-users] Failed to build gem native extension In-Reply-To: References: <6959e1680907252122h19f7c0f0y2a5f47ab9f911bbd@mail.gmail.com> <6959e1680907252228m5c365d33p841fcfd4a43c2032@mail.gmail.com> Message-ID: <6959e1680907252332r40bc0e17k5ec5b44a64ca3537@mail.gmail.com> On Sat, Jul 25, 2009 at 10:46 PM, Jeffrey Roberts wrote: > Hi Aaron, > > Thanks again for your response, I appreciate it greatly, I am like a first > mate on this box, with the captain being out of town, I will def pass on the > idea of upgrading to him when he gets back, but if I can get this working > before then, it reflect well on me =] > > I ran the gem install nokogiri -v 1.3.1 and I get this error > > [root at swn nokogiri]# gem install nokogiri -v 1.3.1 > Building native extensions.? This could take a while... > ERROR:? Error installing nokogiri: > ??????? ERROR: Failed to build gem native extension. > > /usr/bin/ruby extconf.rb > checking for #include > ... no > iconv is missing.? try 'port install iconv' or 'yum install iconv' > > I did a search for iconv.h and I found it here > > /usr/i686-pc-mingw32/sys-root/mingw/include/iconv.h > /usr/include/iconv.h > > any ideas on how I can get the install to recognize that iconv.h ? =] It should Just Work. If you look in your gem installation directory, you should be able to find an mkmf.log file (for nokogiri 1.3.1). Please make that file available for me somewhere. It contains a log of compilation attempts. It should give me a better idea what's wrong. -- Aaron Patterson http://tenderlovemaking.com/ From aaron.patterson at gmail.com Sun Jul 26 03:25:58 2009 From: aaron.patterson at gmail.com (Aaron Patterson) Date: Sun, 26 Jul 2009 00:25:58 -0700 Subject: [Mechanize-users] Failed to build gem native extension In-Reply-To: References: <6959e1680907252122h19f7c0f0y2a5f47ab9f911bbd@mail.gmail.com> <6959e1680907252228m5c365d33p841fcfd4a43c2032@mail.gmail.com> <6959e1680907252332r40bc0e17k5ec5b44a64ca3537@mail.gmail.com> Message-ID: <6959e1680907260025u6c00be0cn8c12265ef1859447@mail.gmail.com> On Sun, Jul 26, 2009 at 12:01 AM, Jeffrey Roberts wrote: > Hello Aaron, > > Thank you for your patience with this, here is what was in the mkmf.log No problem. Looks like you're actually missing "gnu/stubs-32.h". I found this link: http://ubuntuforums.org/archive/index.php/t-370114.html Which indicates that you should do: sudo apt-get install libc6-dev-i386 Do that, then try installing nokogiri. -- Aaron Patterson http://tenderlovemaking.com/ From wonderdog at gmail.com Mon Jul 27 22:11:06 2009 From: wonderdog at gmail.com (Wonderdog) Date: Mon, 27 Jul 2009 19:11:06 -0700 Subject: [Mechanize-users] ignore ssl verification failures? In-Reply-To: <618c07250907242012u4e5d96f7oe08fbd8b5f2354fd@mail.gmail.com> References: <618c07250907242012u4e5d96f7oe08fbd8b5f2354fd@mail.gmail.com> Message-ID: Rockin, man. Worked flawlessly. Thanks! On Fri, Jul 24, 2009 at 8:12 PM, Mike Dalessio wrote: > This should work: > > ??? agent = WWW::Mechanize.new > ??? agent.verify_callback = Proc.new do |ok,x509| > ????? status = x509.error > ????? msg = x509.error_string > ????? logger.warn "server certificate verify: status: #{status}, msg: > #{msg}" if status != 0 > ????? true # this has the side effect of ignoring errors. nice! > ??? end > From jhuizingh at mindless.com Tue Jul 28 21:03:18 2009 From: jhuizingh at mindless.com (Jonathan Huizingh) Date: Tue, 28 Jul 2009 20:03:18 -0500 Subject: [Mechanize-users] Fwd: Iframes in pages In-Reply-To: References: Message-ID: I'm still hoping for an answer to this one. Is there any way to access pages within iframes using mechanize? Jonathan Huizingh Pronunciation: HI-zing ---------- Forwarded message ---------- From: Jonathan Huizingh Date: Fri, Jul 10, 2009 at 6:33 PM Subject: Iframes in pages To: Ruby Mechanize Users List Does mechanize handle iframes that show up within pages that it loads? I have a page that loads a form I need to fill out within an iframe. Is it possible to do something like this? : ? outer_page = agent.get('http://path.to.site.come/outer_page.html') ? form = outer_page.iframes[0].forms[0] ? .... more code ... Is there some way to do something like that? ?If not, is there a recommended way to handle forms embedded in iframes? Thanks, Jonathan Huizingh Pronunciation: ?HI-zing From astarr at wiredquote.com Tue Jul 28 22:33:36 2009 From: astarr at wiredquote.com (Aaron Starr) Date: Tue, 28 Jul 2009 19:33:36 -0700 Subject: [Mechanize-users] Fwd: Iframes in pages In-Reply-To: References: Message-ID: <669cc1ca0907281933g3890503bnd511652b64669c03@mail.gmail.com> How about: outer_page = agent.get('http://path.to.site.come/outer_page.html')frame_page = agent.get(outer_page.frames[0].src) # OR: frame_page = agent.get(outer_page.frame('oFrame').src) form = frame_page.forms[0] On Tue, Jul 28, 2009 at 6:03 PM, Jonathan Huizingh wrote: > I'm still hoping for an answer to this one. Is there any way to > access pages within iframes using mechanize? > > Jonathan Huizingh > Pronunciation: HI-zing > > > > > ---------- Forwarded message ---------- > From: Jonathan Huizingh > Date: Fri, Jul 10, 2009 at 6:33 PM > Subject: Iframes in pages > To: Ruby Mechanize Users List > > > Does mechanize handle iframes that show up within pages that it loads? > > I have a page that loads a form I need to fill out within an iframe. > Is it possible to do something like this? : > > outer_page = agent.get('http://path.to.site.come/outer_page.html') > form = outer_page.iframes[0].forms[0] > .... more code ... > > Is there some way to do something like that? If not, is there a > recommended way to handle forms embedded in iframes? > > Thanks, > > Jonathan Huizingh > Pronunciation: HI-zing > _______________________________________________ > Mechanize-users mailing list > Mechanize-users at rubyforge.org > http://rubyforge.org/mailman/listinfo/mechanize-users > -------------- next part -------------- An HTML attachment was scrubbed... URL: