From radek.simcik at gmail.com Sat Sep 8 13:03:28 2012 From: radek.simcik at gmail.com (Radek Simcik) Date: Sat, 8 Sep 2012 23:03:28 +1000 Subject: [Mechanize-users] Can I log in stackoverflow.com using mechanize? Message-ID: Hi I am trying to log into stackoverflow.com using mechanize but I cannot make it work. Is it even possible? Thank you Radek -------------- next part -------------- An HTML attachment was scrubbed... URL: From benmanns at gmail.com Sat Sep 8 23:26:46 2012 From: benmanns at gmail.com (Benjamin Manns) Date: Sat, 8 Sep 2012 19:26:46 -0400 Subject: [Mechanize-users] Can I log in stackoverflow.com using mechanize? In-Reply-To: References: Message-ID: You may be running into trouble if you are trying to connect from an Amazon EC2 instance. StackOverflow blocks requests from IP addresses assigned to Amazon EC2 to prevent scraping abuse. If that's not your issue, some code and any error messages that you are running into would be helpful. Ben On Sat, Sep 8, 2012 at 9:03 AM, Radek Simcik wrote: > Hi > > I am trying to log into stackoverflow.com using mechanize but I cannot > make it work. Is it even possible? > > Thank you > > Radek > > _______________________________________________ > Mechanize-users mailing list > Mechanize-users at rubyforge.org > http://rubyforge.org/mailman/listinfo/mechanize-users > -- Benjamin Manns benmanns at gmail.com (434) 321-8324 -------------- next part -------------- An HTML attachment was scrubbed... URL: From designium at gmail.com Thu Sep 13 05:01:10 2012 From: designium at gmail.com (Chim Kan) Date: Thu, 13 Sep 2012 01:01:10 -0400 Subject: [Mechanize-users] macgem install mechanize Message-ID: Hi, I'm trying to install mechanize as macruby and it gives the following error: $sudo macgem install mechanize ERROR: While executing gem ... (NoMethodError) undefined method `to_str' for # Is there a way around this? * * Chim K. -------------- next part -------------- An HTML attachment was scrubbed... URL: From kenny at kennymeyer.net Thu Sep 13 13:38:39 2012 From: kenny at kennymeyer.net (Kenny) Date: Thu, 13 Sep 2012 09:38:39 -0400 Subject: [Mechanize-users] macgem install mechanize In-Reply-To: References: Message-ID: <8E06430A24CB4850B78039BAAE5ABDDE@gmail.com> Can you run: sudo macgem install --backtrace mechanize and copy/paste the full output here? -- Kenny Meyer http://www.kennymeyer.net @meyerkenny On Thursday, September 13, 2012 at 1:01 AM, Chim Kan wrote: > Hi, > > I'm trying to install mechanize as macruby and it gives the following error: > > $sudo macgem install mechanize > > ERROR: While executing gem ... (NoMethodError) > undefined method `to_str' for # > > Is there a way around this? > > Chim K. > _______________________________________________ > Mechanize-users mailing list > Mechanize-users at rubyforge.org (mailto:Mechanize-users at rubyforge.org) > http://rubyforge.org/mailman/listinfo/mechanize-users From designium at gmail.com Thu Sep 13 14:48:35 2012 From: designium at gmail.com (Chim Kan) Date: Thu, 13 Sep 2012 10:48:35 -0400 Subject: [Mechanize-users] macgem install mechanize In-Reply-To: <8E06430A24CB4850B78039BAAE5ABDDE@gmail.com> References: <8E06430A24CB4850B78039BAAE5ABDDE@gmail.com> Message-ID: So, I tried to install again now and it worked. Kind of weird. Tks! On Thu, Sep 13, 2012 at 9:38 AM, Kenny wrote: > Can you run: > > sudo macgem install --backtrace mechanize > > and copy/paste the full output here? > > -- > Kenny Meyer > http://www.kennymeyer.net > @meyerkenny > > > On Thursday, September 13, 2012 at 1:01 AM, Chim Kan wrote: > > > Hi, > > > > I'm trying to install mechanize as macruby and it gives the following > error: > > > > $sudo macgem install mechanize > > > > ERROR: While executing gem ... (NoMethodError) > > undefined method `to_str' for # > > > > Is there a way around this? > > > > Chim K. > > _______________________________________________ > > Mechanize-users mailing list > > Mechanize-users at rubyforge.org (mailto:Mechanize-users at rubyforge.org) > > http://rubyforge.org/mailman/listinfo/mechanize-users > > > > _______________________________________________ > Mechanize-users mailing list > Mechanize-users at rubyforge.org > http://rubyforge.org/mailman/listinfo/mechanize-users > -------------- next part -------------- An HTML attachment was scrubbed... URL: From godfreykfc at gmail.com Thu Sep 13 16:47:47 2012 From: godfreykfc at gmail.com (Godfrey Chan) Date: Thu, 13 Sep 2012 09:47:47 -0700 Subject: [Mechanize-users] macgem install mechanize In-Reply-To: References: <8E06430A24CB4850B78039BAAE5ABDDE@gmail.com> Message-ID: Hmm, so I tried this out just to see what's gonna happen, and I got this: mbp:godfrey ~ $ sudo macgem install --backtrace mechanize ERROR: While executing gem ... (TypeError) TypeError /Library/Frameworks/MacRuby.framework/Versions/0.12/usr/lib/ruby/1.9.2/stringio.rb:140:in `read' /Library/Frameworks/MacRuby.framework/Versions/0.12/usr/lib/ruby/1.9.2/stringio.rb:166:in `read_nonblock' /usr/local/bin/macgem:21:in `
' mbp:godfrey ~ $ sudo macgem install mechanize ERROR: While executing gem ... (NoMethodError) undefined method `to_str' for # mbp:godfrey ~ $ sudo macgem install --backtrace mechanize ERROR: While executing gem ... (TypeError) class Gem::Specification needs to have method `_load' /usr/local/bin/macgem:21:in `
' mbp:godfrey ~ $ sudo macgem install --backtrace mechanize Fetching: net-http-digest_auth-1.2.1.gem (100%) Fetching: net-http-persistent-2.7.gem (100%) Fetching: mime-types-1.19.gem (100%) Fetching: nokogiri-1.5.5.gem (100%) Building native extensions. This could take a while... Fetching: ntlm-http-0.1.1.gem (100%) WARNING: ntlm-http-0.1.1 has an invalid nil value for @cert_chain Fetching: webrobots-0.0.13.gem (100%) Fetching: unf_ext-0.0.5.gem (100%) Building native extensions. This could take a while... Fetching: unf-0.0.5.gem (100%) Fetching: domain_name-0.5.3.gem (100%) Fetching: mechanize-2.5.1.gem (100%) Successfully installed net-http-digest_auth-1.2.1 Successfully installed net-http-persistent-2.7 Successfully installed mime-types-1.19 Successfully installed nokogiri-1.5.5 Successfully installed ntlm-http-0.1.1 Successfully installed webrobots-0.0.13 Successfully installed unf_ext-0.0.5 Successfully installed unf-0.0.5 Successfully installed domain_name-0.5.3 Successfully installed mechanize-2.5.1 10 gems installed mbp:godfrey ~ $ macruby is like a box of magic! Godfrey On 2012-09-13, at 7:48 AM, Chim Kan wrote: > So, I tried to install again now and it worked. Kind of weird. Tks! > > On Thu, Sep 13, 2012 at 9:38 AM, Kenny wrote: > Can you run: > > sudo macgem install --backtrace mechanize > > and copy/paste the full output here? > > -- > Kenny Meyer > http://www.kennymeyer.net > @meyerkenny > > > On Thursday, September 13, 2012 at 1:01 AM, Chim Kan wrote: > > > Hi, > > > > I'm trying to install mechanize as macruby and it gives the following error: > > > > $sudo macgem install mechanize > > > > ERROR: While executing gem ... (NoMethodError) > > undefined method `to_str' for # > > > > Is there a way around this? > > > > Chim K. > > _______________________________________________ > > Mechanize-users mailing list > > Mechanize-users at rubyforge.org (mailto:Mechanize-users at rubyforge.org) > > http://rubyforge.org/mailman/listinfo/mechanize-users > > > > _______________________________________________ > Mechanize-users mailing list > Mechanize-users at rubyforge.org > http://rubyforge.org/mailman/listinfo/mechanize-users > > _______________________________________________ > Mechanize-users mailing list > Mechanize-users at rubyforge.org > http://rubyforge.org/mailman/listinfo/mechanize-users -------------- next part -------------- An HTML attachment was scrubbed... URL: From designium at gmail.com Thu Sep 13 17:42:49 2012 From: designium at gmail.com (Chim Kan) Date: Thu, 13 Sep 2012 13:42:49 -0400 Subject: [Mechanize-users] macgem install mechanize In-Reply-To: References: <8E06430A24CB4850B78039BAAE5ABDDE@gmail.com> Message-ID: It happened to me the same exact thing. However, I'm getting this warning from Nokogiri: [50][chimkan: /Users/chimkan/Core/sources/experiments/Hello]$ macirb irb(main):001:0> require 'mechanize' WARNING: Nokogiri was built against LibXML version 2.8.0, but has dynamically loaded 2.7.8 => true irb(main):002:0> It takes 2 seconds to load Nokogiri. I googled for this error, but it's not clear what's happening. Then I loaded this code https://gist.github.com/3716112 : [54][chimkan: /Users/chimkan/Core/sources/experiments/Hello]$ macirb irb(main):001:0> require 'mechanize' WARNING: Nokogiri was built against LibXML version 2.8.0, but has dynamically loaded 2.7.8 => true irb(main):002:0> keyword = 'mixer' => "mixer" irb(main):003:0> @target_url = " http://www.canadiantire.ca/search/search_results.jsp?bmForm=form_endeca_search&bmFormID=1343016irb(main):003:0> @target_url = " http://www.canadiantire.ca/search/search_results.jsp?bmForm=form_endeca_search&bmFormID=1343016788618&bmUID=1343016788618&bmIsForm=true&bmText=quick_search_term&quick_search_term=#{keyword}&bmSubmit=quick_search " => " http://www.canadiantire.ca/search/search_results.jsp?bmForm=form_endeca_search&bmFormID=1343016788618&bmUID=1343016788618&bmIsForm=true&bmText=quick_search_term&quick_search_term=mixer&bmSubmit=quick_search " irb(main):004:0> @agent = Mechanize.new => # nil> irb(main):005:0> @agent.get(@target_url) *Segmentation fault: 11* [55][chimkan: /Users/chimkan/Core/sources/experiments/Hello]$ I also tried googling "segmentation fault: 11" but I can't find anything. Is something wrong with Nokogiri or mechanize? Cheers, On Thu, Sep 13, 2012 at 12:47 PM, Godfrey Chan wrote: > Hmm, so I tried this out just to see what's gonna happen, and I got this: > > mbp:godfrey ~ $ sudo macgem install --backtrace mechanize > ERROR: While executing gem ... (TypeError) > TypeError > /Library/Frameworks/MacRuby.framework/Versions/0.12/usr/lib/ruby/1.9.2/stringio.rb:140:in > `read' > /Library/Frameworks/MacRuby.framework/Versions/0.12/usr/lib/ruby/1.9.2/stringio.rb:166:in > `read_nonblock' > /usr/local/bin/macgem:21:in `
' > > mbp:godfrey ~ $ sudo macgem install mechanize > ERROR: While executing gem ... (NoMethodError) > undefined method `to_str' for # > > mbp:godfrey ~ $ sudo macgem install --backtrace mechanize > ERROR: While executing gem ... (TypeError) > class Gem::Specification needs to have method `_load' > /usr/local/bin/macgem:21:in `
' > > mbp:godfrey ~ $ sudo macgem install --backtrace mechanize > Fetching: net-http-digest_auth-1.2.1.gem (100%) > Fetching: net-http-persistent-2.7.gem (100%) > Fetching: mime-types-1.19.gem (100%) > Fetching: nokogiri-1.5.5.gem (100%) > Building native extensions. This could take a while... > Fetching: ntlm-http-0.1.1.gem (100%) > WARNING: ntlm-http-0.1.1 has an invalid nil value for @cert_chain > Fetching: webrobots-0.0.13.gem (100%) > Fetching: unf_ext-0.0.5.gem (100%) > Building native extensions. This could take a while... > Fetching: unf-0.0.5.gem (100%) > Fetching: domain_name-0.5.3.gem (100%) > Fetching: mechanize-2.5.1.gem (100%) > Successfully installed net-http-digest_auth-1.2.1 > Successfully installed net-http-persistent-2.7 > Successfully installed mime-types-1.19 > Successfully installed nokogiri-1.5.5 > Successfully installed ntlm-http-0.1.1 > Successfully installed webrobots-0.0.13 > Successfully installed unf_ext-0.0.5 > Successfully installed unf-0.0.5 > Successfully installed domain_name-0.5.3 > Successfully installed mechanize-2.5.1 > 10 gems installed > > mbp:godfrey ~ $ > > macruby is like a box of magic! > > Godfrey > > On 2012-09-13, at 7:48 AM, Chim Kan wrote: > > So, I tried to install again now and it worked. Kind of weird. Tks! > > On Thu, Sep 13, 2012 at 9:38 AM, Kenny wrote: > >> Can you run: >> >> sudo macgem install --backtrace mechanize >> >> and copy/paste the full output here? >> >> -- >> Kenny Meyer >> http://www.kennymeyer.net >> @meyerkenny >> >> >> On Thursday, September 13, 2012 at 1:01 AM, Chim Kan wrote: >> >> > Hi, >> > >> > I'm trying to install mechanize as macruby and it gives the following >> error: >> > >> > $sudo macgem install mechanize >> > >> > ERROR: While executing gem ... (NoMethodError) >> > undefined method `to_str' for # >> > >> > Is there a way around this? >> > >> > Chim K. >> > _______________________________________________ >> > Mechanize-users mailing list >> > Mechanize-users at rubyforge.org (mailto:Mechanize-users at rubyforge.org) >> > http://rubyforge.org/mailman/listinfo/mechanize-users >> >> >> >> _______________________________________________ >> Mechanize-users mailing list >> Mechanize-users at rubyforge.org >> http://rubyforge.org/mailman/listinfo/mechanize-users >> > > _______________________________________________ > Mechanize-users mailing list > Mechanize-users at rubyforge.org > http://rubyforge.org/mailman/listinfo/mechanize-users > > > > _______________________________________________ > Mechanize-users mailing list > Mechanize-users at rubyforge.org > http://rubyforge.org/mailman/listinfo/mechanize-users > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jack at groundbreakingsoftware.com Sat Sep 22 18:56:16 2012 From: jack at groundbreakingsoftware.com (Jack Royal-Gordon) Date: Sat, 22 Sep 2012 11:56:16 -0700 Subject: [Mechanize-users] Mechanize and Thread-Safety Revisited Message-ID: This post from Aaron Patterson back in 2009 seems to be pretty definite about how to use Mechanize in a thread-safe way, but I thought I would revisit the question because there are several posts on StackOverflow since then that would seem to contradict it (of course, I trust Aaron's answer more than random answers on StackOverlfow, but just in case I thought I would confirm it). It is still the case that if you do not share agent instances across threads then Mechanize IS thread-safe? Are there any other caveats to be aware of? -------------- next part -------------- An HTML attachment was scrubbed... URL: From drbrain at segment7.net Mon Sep 24 17:53:41 2012 From: drbrain at segment7.net (Eric Hodel) Date: Mon, 24 Sep 2012 10:53:41 -0700 Subject: [Mechanize-users] Mechanize and Thread-Safety Revisited In-Reply-To: References: Message-ID: On Sep 22, 2012, at 11:56 AM, Jack Royal-Gordon wrote: > This post from Aaron Patterson back in 2009 seems to be pretty definite about how to use Mechanize in a thread-safe way Which post? You must include the URL. > I thought I would revisit the question because there are several posts on StackOverflow since then that would seem to contradict it (of course, I trust Aaron's answer more than random answers on StackOverlfow, but just in case I thought I would confirm it). Mechanize is not thread-safe. At least the history and the cookie jar features would require mutual exclusion for safe behavior. > It is still the case that if you do not share agent instances across threads then Mechanize IS thread-safe? Are there any other caveats to be aware of? Separate Mechanize instances don't share state with each other, so one instance per thread is fine. By default mechanize caches the most recent 50 pages visited per instance. For large responses this consumes file descriptors, for small responses this consumes memory. If you're using many threads you will want to reduce the size of history or adjust the response buffering appropriately. From astarr at wiredquote.com Mon Sep 24 19:58:37 2012 From: astarr at wiredquote.com (Aaron Starr) Date: Mon, 24 Sep 2012 12:58:37 -0700 Subject: [Mechanize-users] Mechanize and Thread-Safety Revisited In-Reply-To: References: Message-ID: FYI, this is the post the OP linked to: http://rubyforge.org/pipermail/mechanize-users/2009-September/000449.html Thanks for the tip about history. FWIW, we've been using one agent per thread for a few years now, and it seems to be working well. (As both you, Eric, and Aaron Patterson said it would.) Best, Aaron On Mon, Sep 24, 2012 at 10:53 AM, Eric Hodel wrote: > On Sep 22, 2012, at 11:56 AM, Jack Royal-Gordon < > jack at groundbreakingsoftware.com> wrote: > > This post from Aaron Patterson back in 2009 seems to be pretty definite > about how to use Mechanize in a thread-safe way > > Which post? You must include the URL. > > > I thought I would revisit the question because there are several posts > on StackOverflow since then that would seem to contradict it (of course, I > trust Aaron's answer more than random answers on StackOverlfow, but just in > case I thought I would confirm it). > > Mechanize is not thread-safe. At least the history and the cookie jar > features would require mutual exclusion for safe behavior. > > > It is still the case that if you do not share agent instances across > threads then Mechanize IS thread-safe? Are there any other caveats to be > aware of? > > Separate Mechanize instances don't share state with each other, so one > instance per thread is fine. By default mechanize caches the most recent > 50 pages visited per instance. For large responses this consumes file > descriptors, for small responses this consumes memory. If you're using > many threads you will want to reduce the size of history or adjust the > response buffering appropriately. > _______________________________________________ > Mechanize-users mailing list > Mechanize-users at rubyforge.org > http://rubyforge.org/mailman/listinfo/mechanize-users > -------------- next part -------------- An HTML attachment was scrubbed... URL: From neil at cozyspace.net Wed Sep 26 17:43:25 2012 From: neil at cozyspace.net (Neil Chambers) Date: Wed, 26 Sep 2012 18:43:25 +0100 Subject: [Mechanize-users] accessing unnamed element Message-ID: <50633EBD.7010201@cozyspace.net> Hello Mechanizers, I am using Mechanize 2.5.1 and am trying to interact with some non-form input (it's probably an event driven ajax submission). I want to add some input into the textarea with the id of 'comment' and click the button with the id of 'button'. Try as I might, I simply can't get a handle on the elements. The HTML looks like this:

I can, of course, get the form object but selection of the non-form elements is escaping me. Any ideas? Many thanks, Neil -------------- next part -------------- An HTML attachment was scrubbed... URL: