From hunkybill at gmail.com Tue May 4 15:28:05 2010 From: hunkybill at gmail.com (Dave Lazar) Date: Tue, 4 May 2010 15:28:05 -0400 Subject: [Mechanize-users] Problems with get command Message-ID: Hi, I am logging in to one site and experiencing no trouble getting a form filled in a clicking a submit. I am trying a second site now, and it is not working. I am getting a 403 => Net::HTTPForbidden /Library/Ruby/Gems/1.8/gems/mechanize-1.0.0/lib/mechanize.rb:259:in `get' The page is not protected as it is a login page I am doing the get command on. agent = Mechanize.new page = agent.get("http://www.tirerackwholesale.com/whlogin/Login.jsp") pp page I am not sure why this page is returning a 403? Is there a convenient way to get more information about the exact cause of the get failure?? Thanks -------------- next part -------------- An HTML attachment was scrubbed... URL: From miha.ploha at gmail.com Tue May 4 16:13:51 2010 From: miha.ploha at gmail.com (Mihael) Date: Tue, 4 May 2010 22:13:51 +0200 Subject: [Mechanize-users] Problems with get command In-Reply-To: References: Message-ID: <1F5871BA-6F83-4786-8445-D80A2214FAC7@gmail.com> hey, you need to set the user agent: a = Mechanize.new { |agent| agent.user_agent_alias = 'Mac Safari'} page = a.get("http://www.tirerackwholesale.com/whlogin/Login.jsp") On May 4, 2010, at 9:28 PM, Dave Lazar wrote: > Hi, > > I am logging in to one site and experiencing no trouble getting a form filled in a clicking a submit. > > I am trying a second site now, and it is not working. I am getting a 403 => Net::HTTPForbidden > /Library/Ruby/Gems/1.8/gems/mechanize-1.0.0/lib/mechanize.rb:259:in `get' > > The page is not protected as it is a login page I am doing the get command on. > > agent = Mechanize.new > page = agent.get("http://www.tirerackwholesale.com/whlogin/Login.jsp") > pp page > > I am not sure why this page is returning a 403? > > Is there a convenient way to get more information about the exact cause of the get failure?? > > Thanks > > > _______________________________________________ > Mechanize-users mailing list > Mechanize-users at rubyforge.org > http://rubyforge.org/mailman/listinfo/mechanize-users -------------- next part -------------- An HTML attachment was scrubbed... URL: From hunkybill at gmail.com Tue May 4 16:23:52 2010 From: hunkybill at gmail.com (Dave Lazar) Date: Tue, 4 May 2010 16:23:52 -0400 Subject: [Mechanize-users] Problems with get command In-Reply-To: <1F5871BA-6F83-4786-8445-D80A2214FAC7@gmail.com> References: <1F5871BA-6F83-4786-8445-D80A2214FAC7@gmail.com> Message-ID: Hi, Awesome! Thanks!! I did not suspect the User Agent would result in that problem. Dave On Tue, May 4, 2010 at 4:13 PM, Mihael wrote: > hey, > > you need to set the user agent: > > a = Mechanize.new { |agent| agent.user_agent_alias = 'Mac Safari'} > page = a.get("http://www.tirerackwholesale.com/whlogin/Login.jsp") > > > On May 4, 2010, at 9:28 PM, Dave Lazar wrote: > > Hi, > > I am logging in to one site and experiencing no trouble getting a form > filled in a clicking a submit. > > I am trying a second site now, and it is not working. I am getting a 403 => > Net::HTTPForbidden > /Library/Ruby/Gems/1.8/gems/mechanize-1.0.0/lib/mechanize.rb:259:in `get' > > The page is not protected as it is a login page I am doing the get command > on. > > agent = Mechanize.new > page = agent.get("http://www.tirerackwholesale.com/whlogin/Login.jsp") > pp page > > I am not sure why this page is returning a 403? > > Is there a convenient way to get more information about the exact cause of > the get failure?? > > Thanks > > > _______________________________________________ > Mechanize-users mailing list > Mechanize-users at rubyforge.org > http://rubyforge.org/mailman/listinfo/mechanize-users > > > > _______________________________________________ > Mechanize-users mailing list > Mechanize-users at rubyforge.org > http://rubyforge.org/mailman/listinfo/mechanize-users > -------------- next part -------------- An HTML attachment was scrubbed... URL: From icambron at alum.mit.edu Fri May 7 01:14:06 2010 From: icambron at alum.mit.edu (Isaac Cambron) Date: Fri, 7 May 2010 01:14:06 -0400 Subject: [Mechanize-users] mechanize and eventmachine Message-ID: I'm planning to process a bunch of pages in parallel, and I'd love to do it with fibers and asynchronous web requests, something like http://www.igvita.com/2010/03/22/untangling-evented-code-with-ruby-fibers/. It looks like Mechanize is built around Net::HTTP, which AFAIK, is synchronous only. Is there a way of mixing Eventmachine with Mechanize, or is it too closely tied into Net::HTTP? I'm diving into code, but I'm wondering if a) anyone's tried this, or maybe b) it's just crazy talk and I should use threads or something, or even c) I'm just totally missing some larger point. Thanks, Isaac -------------- next part -------------- An HTML attachment was scrubbed... URL: From mike at csa.net Fri May 7 09:04:16 2010 From: mike at csa.net (Mike Dalessio) Date: Fri, 7 May 2010 09:04:16 -0400 Subject: [Mechanize-users] mechanize and eventmachine In-Reply-To: References: Message-ID: On Fri, May 7, 2010 at 1:14 AM, Isaac Cambron wrote: > I'm planning to process a bunch of pages in parallel, and I'd love to do it > with fibers and asynchronous web requests, something like > http://www.igvita.com/2010/03/22/untangling-evented-code-with-ruby-fibers/. It > looks like Mechanize is built around Net::HTTP, which AFAIK, is synchronous > only. Is there a way of mixing Eventmachine with Mechanize, or is it too > closely tied into Net::HTTP? I'm diving into code, but I'm wondering if a) > anyone's tried this, or maybe b) it's just crazy talk and I should use > threads or something, or even c) I'm just totally missing some larger point. > > You should look at Paul Dix's typhoeus, http://github.com/pauldix/typhoeus. > Thanks, > Isaac > > _______________________________________________ > Mechanize-users mailing list > Mechanize-users at rubyforge.org > http://rubyforge.org/mailman/listinfo/mechanize-users > -------------- next part -------------- An HTML attachment was scrubbed... URL: From icambron at alum.mit.edu Mon May 10 17:49:52 2010 From: icambron at alum.mit.edu (Isaac Cambron) Date: Mon, 10 May 2010 17:49:52 -0400 Subject: [Mechanize-users] mechanize and eventmachine In-Reply-To: References: Message-ID: Thanks, I'll check that out. It looks like the right approach for me is to log in with mechanize, manually add the cookies to Typhoeus, make my requests, and then do complex parsing with Nokigiri. Is that a non-crazy approach? On Fri, May 7, 2010 at 9:04 AM, Mike Dalessio wrote: > > > On Fri, May 7, 2010 at 1:14 AM, Isaac Cambron wrote: > >> I'm planning to process a bunch of pages in parallel, and I'd love to do >> it with fibers and asynchronous web requests, something like >> http://www.igvita.com/2010/03/22/untangling-evented-code-with-ruby-fibers/. It >> looks like Mechanize is built around Net::HTTP, which AFAIK, is synchronous >> only. Is there a way of mixing Eventmachine with Mechanize, or is it too >> closely tied into Net::HTTP? I'm diving into code, but I'm wondering if a) >> anyone's tried this, or maybe b) it's just crazy talk and I should use >> threads or something, or even c) I'm just totally missing some larger point. >> >> > You should look at Paul Dix's typhoeus, http://github.com/pauldix/typhoeus > . > > >> Thanks, >> Isaac >> >> _______________________________________________ >> Mechanize-users mailing list >> Mechanize-users at rubyforge.org >> http://rubyforge.org/mailman/listinfo/mechanize-users >> > > > _______________________________________________ > Mechanize-users mailing list > Mechanize-users at rubyforge.org > http://rubyforge.org/mailman/listinfo/mechanize-users > -------------- next part -------------- An HTML attachment was scrubbed... URL: From overco at me.com Mon May 31 11:10:17 2010 From: overco at me.com (overco at me.com) Date: Mon, 31 May 2010 11:10:17 -0400 Subject: [Mechanize-users] Mechanize SelectList !?! Message-ID: Hi there, Firstly, thanks for Mechanize! I?m finding it incredibly useful so far. The issue I?m stuck on is to do with select lists, I can?t seem to figure out how they should be accessed and selected. Here is the HTML:
I?ve been trying to the use the following pieces of Ruby code to change the options in the 2nd list: agent.page.forms[0].set_fields(:interval => "last7days") page = agent.submit(agent.page.forms[0]) And: agent.page.forms[0].interval = ?last7days? page = agent.submit(agent.page.forms[0]) Both of these fail to work as I?m expecting. Can anyone suggest some tips or some examples of how to use correctly use select boxes with Mechanize? Best Regards, Jonathan -------------- next part -------------- An HTML attachment was scrubbed... URL: From justinbrinkerhoff at gmail.com Mon May 31 13:00:45 2010 From: justinbrinkerhoff at gmail.com (Justin Brinkerhoff) Date: Mon, 31 May 2010 11:00:45 -0600 Subject: [Mechanize-users] Mechanize SelectList !?! In-Reply-To: References: Message-ID: Have you taken a look at the API? http://mechanize.rubyforge.org/mechanize/Mechanize/Form/SelectList.html However, the documentation is not very intuitive. If you want to select a specific option of a dropdown on a webpage, you have to use the argument .select. Let's say I want to go to JimBeam.com. The landing page has an age verification form. I could use Mechanize to do this for me. So I open up irb, and do the following: > irb > require 'rubygems' > require 'mechanize' > agent = Mechanize.new > agent.get("http://www.jimbeam.com/lpa") > form = agent.page.forms.first > year = form.field("year") # Year select list. > month = form.field("month") # Month select list > day = form.field("day") # Day select list Ok, now that I have all my variables in place for each of the lists, now I want to tell it to enter my birthdate. Being my birthdate is 3/6/84, I would do the following: > year.option_with(:value => /1984/).select # Changes the selected value to 1984 > month.option_with(:value => /3/).select # Changes selected value to 3 > day.option_with(:value => /6/).select # Changes selected value to 6 Now, I want to submit the form with the new values. So, I'll do the standard form.submit using my form variable. > form.submit And there you have it. So ultimately, you want to find the selectlist to control, then find the option, and then use the .select method to change the value. I hope this helps. Thanks, Justin On Mon, May 31, 2010 at 9:10 AM, wrote: > Hi there, > > Firstly, thanks for Mechanize! I?m finding it incredibly useful so far. > > The issue I?m stuck on is to do with select lists, I can?t seem to figure > out how they should be accessed and selected. > > Here is the HTML: > > *
* > * *** > * *** > * *** > * *** > * *** > * **
* > * *** > * *** > * *** > * **
* > *
* > > I?ve been trying to the use the following pieces of Ruby code to change the > options in the 2nd list: > > *agent.page.forms[0].set_fields(:interval => "last7days")* > > *page = agent.submit(agent.page.forms[0])* > > > And: > > *agent.page.forms[0].interval = ?last7days?* > > *page = agent.submit(agent.page.forms[0])* > > > Both of these fail to work as I?m expecting. Can anyone suggest some tips > or some examples of how to use correctly use select boxes with Mechanize? > > Best Regards, > Jonathan > > > > > > > > _______________________________________________ > Mechanize-users mailing list > Mechanize-users at rubyforge.org > http://rubyforge.org/mailman/listinfo/mechanize-users > -------------- next part -------------- An HTML attachment was scrubbed... URL: From trimolesi at gmail.com Mon May 31 13:34:13 2010 From: trimolesi at gmail.com (Thiago Imolesi) Date: Mon, 31 May 2010 14:34:13 -0300 Subject: [Mechanize-users] Log in orkut Message-ID: Well , I need to log in and retrieve data from a community on orkut, but still can not log in, someone has done something similar ? Thanks -- Att. Thiago Imolesi -------------- next part -------------- An HTML attachment was scrubbed... URL: From overco at me.com Mon May 31 13:35:28 2010 From: overco at me.com (overco at me.com) Date: Mon, 31 May 2010 13:35:28 -0400 Subject: [Mechanize-users] Testing? Message-ID: Hi there, Just wanted to test to see if my messages are getting posted to the list? From trimolesi at gmail.com Mon May 31 13:39:34 2010 From: trimolesi at gmail.com (Thiago Imolesi) Date: Mon, 31 May 2010 14:39:34 -0300 Subject: [Mechanize-users] Testing? In-Reply-To: References: Message-ID: Don't arrived On Mon, May 31, 2010 at 14:35, wrote: > Hi there, > > Just wanted to test to see if my messages are getting posted to the list? > _______________________________________________ > Mechanize-users mailing list > Mechanize-users at rubyforge.org > http://rubyforge.org/mailman/listinfo/mechanize-users > -- Att. Thiago Imolesi -------------- next part -------------- An HTML attachment was scrubbed... URL: From overco at me.com Mon May 31 15:10:04 2010 From: overco at me.com (overco at me.com) Date: Mon, 31 May 2010 15:10:04 -0400 Subject: [Mechanize-users] Mechanize SelectList !?! In-Reply-To: References: Message-ID: <88A1FCE6-49B9-4325-A57E-67175E99986A@me.com> Thanks for forwarding. I didn?t receive your message earlier, I don?t think the list is working for me. I understand, what your suggesting below and have tried to implement it, however, still no joy. Does Mechanize have trouble with AJAX forms? I think the one I?m trying to manipulate might be AJAX because it doesn?t refresh or redirect the page. On 31 May 2010, at 14:35, Justin Brinkerhoff wrote: > Not sure if you got my reply or not earlier, due to your later email asking if we are getting your messages on the list. > > So here is my forwarded response in the event you didn't get it. > > ---------- Forwarded message ---------- > From: Justin Brinkerhoff > Date: Mon, May 31, 2010 at 11:00 AM > Subject: Re: [Mechanize-users] Mechanize SelectList !?! > To: Ruby Mechanize Users List > > > Have you taken a look at the API? > > http://mechanize.rubyforge.org/mechanize/Mechanize/Form/SelectList.html > > However, the documentation is not very intuitive. > > If you want to select a specific option of a dropdown on a webpage, you have to use the argument .select. > > Let's say I want to go to JimBeam.com. The landing page has an age verification form. > > I could use Mechanize to do this for me. > > So I open up irb, and do the following: > > > irb > > require 'rubygems' > > require 'mechanize' > > agent = Mechanize.new > > agent.get("http://www.jimbeam.com/lpa") > > form = agent.page.forms.first > > year = form.field("year") # Year select list. > > month = form.field("month") # Month select list > > day = form.field("day") # Day select list > > Ok, now that I have all my variables in place for each of the lists, now I want to tell it to enter my birthdate. Being my birthdate is 3/6/84, I would do the following: > > > year.option_with(:value => /1984/).select # Changes the selected value to 1984 > > month.option_with(:value => /3/).select # Changes selected value to 3 > > day.option_with(:value => /6/).select # Changes selected value to 6 > > Now, I want to submit the form with the new values. So, I'll do the standard form.submit using my form variable. > > > form.submit > > And there you have it. > > So ultimately, you want to find the selectlist to control, then find the option, and then use the .select method to change the value. > > I hope this helps. > > Thanks, > > Justin > > On Mon, May 31, 2010 at 9:10 AM, wrote: > Hi there, > > Firstly, thanks for Mechanize! I?m finding it incredibly useful so far. > > The issue I?m stuck on is to do with select lists, I can?t seem to figure out how they should be accessed and selected. > > Here is the HTML: > >
> > > > > >
> > > >
>
> > I?ve been trying to the use the following pieces of Ruby code to change the options in the 2nd list: > > agent.page.forms[0].set_fields(:interval => "last7days") > page = agent.submit(agent.page.forms[0]) > > And: > > agent.page.forms[0].interval = ?last7days? > page = agent.submit(agent.page.forms[0]) > > Both of these fail to work as I?m expecting. Can anyone suggest some tips or some examples of how to use correctly use select boxes with Mechanize? > > Best Regards, > Jonathan > > > > > > > > _______________________________________________ > Mechanize-users mailing list > Mechanize-users at rubyforge.org > http://rubyforge.org/mailman/listinfo/mechanize-users > > Jonathan Clarke ? Campaign Manager U365 Cell Number: +1-246-232-0770 Skype IM Chat: jonathan.clarke This message is intended only for the addressee and contains privileged and confidential information. If you have received this message in error please notify me immediately and delete the original message and destroy any copies of it. -------------- next part -------------- An HTML attachment was scrubbed... URL: