From peter at rubyrailways.com  Sun May  6 08:30:01 2007
From: peter at rubyrailways.com (Peter Szinek)
Date: Sun, 06 May 2007 14:30:01 +0200
Subject: [Mechanize-users] How to submit this form?
Message-ID: <463DCA49.6030101@rubyrailways.com>

Hey all,

maybe I am missing the obvious again, but I can not submit the form on 
buy.com:

require 'rubygems'
require 'mechanize'

agent = WWW::Mechanize.new
agent.user_agent_alias = 'Mac Safari'
page = agent.get("http://www.buy.com")
search_form = page.forms.with.name("searchbox").first
p search_form.buttons

the last array is empty... OK, probably 'GO' is not a button - but then 
what it is? How to submit such forms in general (which should be 
submitted with an image button, which is not really an image button...)

TIA,
Peter

From barjunk at attglobal.net  Sun May  6 17:00:15 2007
From: barjunk at attglobal.net (barsalou)
Date: Sun, 06 May 2007 13:00:15 -0800
Subject: [Mechanize-users] How to submit this form?
In-Reply-To: <463DCA49.6030101@rubyrailways.com>
References: <463DCA49.6030101@rubyrailways.com>
Message-ID: <20070506130015.i1bjgfw7fsog04w4@lcgalaska.com>

Quoting Peter Szinek <peter at rubyrailways.com>:

> Hey all,
>
> maybe I am missing the obvious again, but I can not submit the form on
> buy.com:
>
> require 'rubygems'
> require 'mechanize'
>
> agent = WWW::Mechanize.new
> agent.user_agent_alias = 'Mac Safari'
> page = agent.get("http://www.buy.com")
> search_form = page.forms.with.name("searchbox").first
> p search_form.buttons
>
> the last array is empty... OK, probably 'GO' is not a button - but then
> what it is? How to submit such forms in general (which should be
> submitted with an image button, which is not really an image button...)

Peter,

My only suggestion is that you are probably going to use 
search_forms.submit.  View the source of the page, I believe the GO! is 
an image.

So fill the search box and use search_form.submit.

Mike B.

----------------------------------------------------------------
This message was sent using IMP, the Internet Messaging Program.


From tanshunquan at gmail.com  Tue May  8 00:32:04 2007
From: tanshunquan at gmail.com (Shunquan Tan)
Date: Tue, 8 May 2007 12:32:04 +0800
Subject: [Mechanize-users] the website ban me when I use mechanize to grab
	its content, how to solve it?
Message-ID: <f1e76a910705072132o340d477bp31161304df3f1d13@mail.gmail.com>

Dear all,

I keep grab photos from one website.

At first I show them the default user agent (WWW-Mechanize). But
obviously they don't like this behavior and ban the user agent
WWW-Mechanize. So I have to forge an IE user agent. It works for some
time. But the website administrator seemed to detect my behavior and
banned me again. This time I can not bypass it by simply modifying the
user agent.

I already set the user-agent of the Mechanize the same as the one my
IE browser uses. But it can't work. I can use IE to browse the
website, but I only get an empty page when I use Mechanize. All the
decisions are done the server side because in my client side any
scripts or redirect header is not received. All I get is an empty
page! So how can they judge I am not using a standard browser and how
can I bypass this obstacle again? Andy suggestions are appreciated.

Thanks in advance,
Harish

From andy at petdance.com  Tue May  8 00:42:56 2007
From: andy at petdance.com (Andy Lester)
Date: Mon, 7 May 2007 23:42:56 -0500
Subject: [Mechanize-users] the website ban me when I use mechanize to
	grab its content, how to solve it?
In-Reply-To: <f1e76a910705072132o340d477bp31161304df3f1d13@mail.gmail.com>
References: <f1e76a910705072132o340d477bp31161304df3f1d13@mail.gmail.com>
Message-ID: <50DCBE31-363A-4A3B-8415-C15186F48939@petdance.com>


On May 7, 2007, at 11:32 PM, Shunquan Tan wrote:

> But the website administrator seemed to detect my behavior and
> banned me again. This time I can not bypass it by simply modifying the
> user agent.

So don't scrape the site.  Mechanize isn't created so you can abuse  
websites.

--
Andy Lester => andy at petdance.com => www.petdance.com => AIM:petdance





From tim.bounceback at gmail.com  Sat May 12 13:27:11 2007
From: tim.bounceback at gmail.com (Timothy van Bremen)
Date: Sat, 12 May 2007 13:27:11 -0400
Subject: [Mechanize-users] Setting form-values
Message-ID: <f5fe12360705121027y6bfddf01ib849eb7c0cbdac50@mail.gmail.com>

Hi,

I've recently had some trouble setting form values in my code - for example:

require 'rubygems'
require 'mechanize'
def save(pagename, content, editsummary = '')
    agent = WWW::Mechanize.new
    agent.user_agent = 'Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1)'
    page = agent.get('http://en.wikipedia.org/w/index.php?title=' + pagename
+ '&action=edit')
    form = page.forms.with.name("editform").first
    form.wpTextbox1 = content
    form.wpSummary = editsummary
    page = agent.submit form
end
save('User:Tim.bounceback/testing', 'bleh')

Yields the following error message:
bleh.rb:8:in `save': undefined method `wpTextbox1=' for nil:NilClass
(NoMethodError)
        from bleh.rb:12
Any help would be appreciated

Thanks!

Tim
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://rubyforge.org/pipermail/mechanize-users/attachments/20070512/83f4907c/attachment.html 

From aaron at tenderlovemaking.com  Sat May 12 22:39:30 2007
From: aaron at tenderlovemaking.com (Aaron Patterson)
Date: Sat, 12 May 2007 19:39:30 -0700
Subject: [Mechanize-users] Setting form-values
In-Reply-To: <f5fe12360705121027y6bfddf01ib849eb7c0cbdac50@mail.gmail.com>
References: <f5fe12360705121027y6bfddf01ib849eb7c0cbdac50@mail.gmail.com>
Message-ID: <20070513023930.GA10046@hasselhoff.lan>

On Sat, May 12, 2007 at 01:27:11PM -0400, Timothy van Bremen wrote:
> Hi,
> 
> I've recently had some trouble setting form values in my code - for example:
> 
> require 'rubygems'
> require 'mechanize'
> def save(pagename, content, editsummary = '')
>    agent = WWW::Mechanize.new
>    agent.user_agent = 'Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1)'
>    page = agent.get('http://en.wikipedia.org/w/index.php?title=' + pagename
> + '&action=edit')
>    form = page.forms.with.name("editform").first
>    form.wpTextbox1 = content
>    form.wpSummary = editsummary
>    page = agent.submit form
> end
> save('User:Tim.bounceback/testing', 'bleh')
> 
> Yields the following error message:
> bleh.rb:8:in `save': undefined method `wpTextbox1=' for nil:NilClass
> (NoMethodError)
>        from bleh.rb:12
> Any help would be appreciated

There doesn't seem to be any form with the name 'editform'.  I verified
that in my browser and with mechanize.

-- 
Aaron Patterson
http://tenderlovemaking.com/

From tim.bounceback at gmail.com  Sun May 13 09:06:50 2007
From: tim.bounceback at gmail.com (Timothy van Bremen)
Date: Sun, 13 May 2007 09:06:50 -0400
Subject: [Mechanize-users] Setting form-values
Message-ID: <f5fe12360705130606q3606a4f2meea10c35047c4f11@mail.gmail.com>

OK, but what's this?

<form id="editform" name="editform" method="post"
action="/w/index.php?title=%27User:Tim.bounceback/testing%27&amp;action=submit"
enctype="multipart/form-data">

It can be found on line 126 (for me).

Tim




-- 
Tim: "So would that be a paradox?"
Andy: "No, that would just create a hole in the space-time continuum."

From aaron at tenderlovemaking.com  Sun May 13 14:00:46 2007
From: aaron at tenderlovemaking.com (Aaron Patterson)
Date: Sun, 13 May 2007 11:00:46 -0700
Subject: [Mechanize-users] Setting form-values
In-Reply-To: <f5fe12360705130606q3606a4f2meea10c35047c4f11@mail.gmail.com>
References: <f5fe12360705130606q3606a4f2meea10c35047c4f11@mail.gmail.com>
Message-ID: <20070513180046.GA21494@hasselhoff.lan>

On Sun, May 13, 2007 at 09:06:50AM -0400, Timothy van Bremen wrote:
> OK, but what's this?
> 
> <form id="editform" name="editform" method="post"
> action="/w/index.php?title=%27User:Tim.bounceback/testing%27&amp;action=submit"
> enctype="multipart/form-data">
> 
> It can be found on line 126 (for me).

Are you logged in to wikipedia in your browser?  I think you have to be
logged in to get that form.  If thats the case, you'd have to get
mechanize to log in to wikipedia before it could see that form.

-- 
Aaron Patterson
http://tenderlovemaking.com/

From farleyknight at gmail.com  Sun May 20 17:12:45 2007
From: farleyknight at gmail.com (Farley Knight)
Date: Sun, 20 May 2007 17:12:45 -0400
Subject: [Mechanize-users] Tips on testing
Message-ID: <c6f9c9e70705201412r7de80b4bu39f78c9b0367d922@mail.gmail.com>

Hey WWW::Mechanize users,

I've been using Mechanize for about 3 or 4 months now, and I'm rather
enjoying it. I've done scraping in PHP & Python, starting with raw text and
HTTP requests, moving up to regex and Curl, and finally landed myself here,
making me much more productive when scraping pages.

At the moment, I'm in the middle of a decent sized project, which requires
scraping lots of pages. Now, I'd like to be able to refactor much of it,
however I'm not entirely happy with the tests that I'm writing. I'm creating
way too many global variables, and monkey patching classes instead of
creating mocks. I would ideally like to have something like ActiveRecord's
fixtures, where I can serialize a page along with some meta-data about it,
along with a mock instance of my Mechanize instance.

Has anyone else done any extensive testing with Mechanize? I'd like to hear
about what kinds of tips and tricks you employed to come up with something
that worked well.

Thanks!

-Rob
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://rubyforge.org/pipermail/mechanize-users/attachments/20070520/fcd240d3/attachment.html 

From mguterl at gmail.com  Wed May 23 14:27:32 2007
From: mguterl at gmail.com (Michael Guterl)
Date: Wed, 23 May 2007 14:27:32 -0400
Subject: [Mechanize-users] Error when posting form
Message-ID: <944a03770705231127y3b786b59n41615c133e65fa02@mail.gmail.com>

Trying to submit a form is resulting in the following error:

>> page = agent.submit(f, f.buttons.first)
query: "action=start&quick=Y&clientApp=0&clientID=MCO"
Net::HTTP::Post: /dealerApply/controller.jsp
using cookie:
JSESSIONID=GJ4NCvGJDTlrvrKsLw17hZ3FnlXFdVkvB5hh1jLkHKcH4Tn27371!-538794652
request-header: accept-language => en-us,en;q0.5
request-header: connection => keep-alive
request-header: accept => */*
request-header: accept-encoding => gzip,identity
request-header: user-agent => WWW-Mechanize/0.6.8 (
http://rubyforge.org/projects/mechanize/)
request-header: content-type => application/x-www-form-urlencoded
request-header: cookie =>
JSESSIONID=GJ4NCvGJDTlrvrKsLw17hZ3FnlXFdVkvB5hh1jLkHKcH4Tn27371!-538794652
request-header: referer => https://censored.com/dealerApply/
request-header: accept-charset => ISO-8859-1,utf-8;q=0.7,*;q=0.7
request-header: content-length => 47
request-header: keep-alive => 300
Errno::EPIPE: Broken pipe
        from /usr/local/lib/ruby/1.8/openssl/buffering.rb:178:in `syswrite'
        from /usr/local/lib/ruby/1.8/openssl/buffering.rb:178:in `do_write'
        from /usr/local/lib/ruby/1.8/openssl/buffering.rb:192:in `write'
        from /usr/local/lib/ruby/1.8/net/protocol.rb:175:in `write0'
        from /usr/local/lib/ruby/1.8/net/protocol.rb:151:in `write'
        from /usr/local/lib/ruby/1.8/net/protocol.rb:166:in `writing'
        from /usr/local/lib/ruby/1.8/net/protocol.rb:150:in `write'
        from /usr/local/lib/ruby/1.8/net/http.rb:1571:in `write_header'
        from /usr/local/lib/ruby/1.8/net/http.rb:1536:in
`send_request_with_body'
        from /usr/local/lib/ruby/1.8/net/http.rb:1522:in `exec'
        from /usr/local/lib/ruby/1.8/net/http.rb:1045:in `request'
        from /usr/local/lib/ruby/gems/1.8/gems/mechanize-0.6.8/lib/mechanize.rb:443:in
`fetch_page'
        from /usr/local/lib/ruby/gems/1.8/gems/mechanize-0.6.8/lib/mechanize.rb:357:in
`post_form'
        from /usr/local/lib/ruby/gems/1.8/gems/mechanize-0.6.8/lib/mechanize.rb:234:in
`submit'
        from (irb):28
        from :0

I'd be happy to provide any other information needed to help debug the
problem.

Thanks,
Michael Guterl
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://rubyforge.org/pipermail/mechanize-users/attachments/20070523/77d79260/attachment.html 

From aaron at tenderlovemaking.com  Wed May 23 17:56:16 2007
From: aaron at tenderlovemaking.com (Aaron Patterson)
Date: Wed, 23 May 2007 14:56:16 -0700
Subject: [Mechanize-users] Tips on testing
In-Reply-To: <c6f9c9e70705201412r7de80b4bu39f78c9b0367d922@mail.gmail.com>
References: <c6f9c9e70705201412r7de80b4bu39f78c9b0367d922@mail.gmail.com>
Message-ID: <20070523215616.GA10770@hasselhoff.lan>

Hi Farley!

On Sun, May 20, 2007 at 05:12:45PM -0400, Farley Knight wrote:
> Hey WWW::Mechanize users,
> 
> I've been using Mechanize for about 3 or 4 months now, and I'm rather
> enjoying it. I've done scraping in PHP & Python, starting with raw text and
> HTTP requests, moving up to regex and Curl, and finally landed myself here,
> making me much more productive when scraping pages.
> 
> At the moment, I'm in the middle of a decent sized project, which requires
> scraping lots of pages. Now, I'd like to be able to refactor much of it,
> however I'm not entirely happy with the tests that I'm writing. I'm creating
> way too many global variables, and monkey patching classes instead of
> creating mocks. I would ideally like to have something like ActiveRecord's
> fixtures, where I can serialize a page along with some meta-data about it,
> along with a mock instance of my Mechanize instance.
> 
> Has anyone else done any extensive testing with Mechanize? I'd like to hear
> about what kinds of tips and tricks you employed to come up with something
> that worked well.

Sorry for taking so long to reply.  Typically what I'll do for testing
is moch out my mechanize objects.  Sometimes I like to think of it as a
web service, and test it using similar patterns.  Unfortunately people's
websites can change, and you'll only really run in to that during live
runs.

I suggest checking out flexmock:

  http://flexmock.rubyforge.org/

Good luck!  Hope that helps.

-- 
Aaron Patterson
http://tenderlovemaking.com/

From aaron at tenderlovemaking.com  Wed May 23 17:57:46 2007
From: aaron at tenderlovemaking.com (Aaron Patterson)
Date: Wed, 23 May 2007 14:57:46 -0700
Subject: [Mechanize-users] Error when posting form
In-Reply-To: <944a03770705231127y3b786b59n41615c133e65fa02@mail.gmail.com>
References: <944a03770705231127y3b786b59n41615c133e65fa02@mail.gmail.com>
Message-ID: <20070523215746.GB10770@hasselhoff.lan>

Hello Michael,

On Wed, May 23, 2007 at 02:27:32PM -0400, Michael Guterl wrote:
> Trying to submit a form is resulting in the following error:
> 
> >>page = agent.submit(f, f.buttons.first)
> query: "action=start&quick=Y&clientApp=0&clientID=MCO"
> Net::HTTP::Post: /dealerApply/controller.jsp
> using cookie:
> JSESSIONID=GJ4NCvGJDTlrvrKsLw17hZ3FnlXFdVkvB5hh1jLkHKcH4Tn27371!-538794652
> request-header: accept-language => en-us,en;q0.5
> request-header: connection => keep-alive
> request-header: accept => */*
> request-header: accept-encoding => gzip,identity
> request-header: user-agent => WWW-Mechanize/0.6.8 (
> http://rubyforge.org/projects/mechanize/)
> request-header: content-type => application/x-www-form-urlencoded
> request-header: cookie =>
> JSESSIONID=GJ4NCvGJDTlrvrKsLw17hZ3FnlXFdVkvB5hh1jLkHKcH4Tn27371!-538794652
> request-header: referer => https://censored.com/dealerApply/
> request-header: accept-charset => ISO-8859-1,utf-8;q=0.7,*;q=0.7
> request-header: content-length => 47
> request-header: keep-alive => 300
> Errno::EPIPE: Broken pipe
>        from /usr/local/lib/ruby/1.8/openssl/buffering.rb:178:in `syswrite'
>        from /usr/local/lib/ruby/1.8/openssl/buffering.rb:178:in `do_write'
>        from /usr/local/lib/ruby/1.8/openssl/buffering.rb:192:in `write'
>        from /usr/local/lib/ruby/1.8/net/protocol.rb:175:in `write0'
>        from /usr/local/lib/ruby/1.8/net/protocol.rb:151:in `write'
>        from /usr/local/lib/ruby/1.8/net/protocol.rb:166:in `writing'
>        from /usr/local/lib/ruby/1.8/net/protocol.rb:150:in `write'
>        from /usr/local/lib/ruby/1.8/net/http.rb:1571:in `write_header'
>        from /usr/local/lib/ruby/1.8/net/http.rb:1536:in
> `send_request_with_body'
>        from /usr/local/lib/ruby/1.8/net/http.rb:1522:in `exec'
>        from /usr/local/lib/ruby/1.8/net/http.rb:1045:in `request'
>        from 
>        /usr/local/lib/ruby/gems/1.8/gems/mechanize-0.6.8/lib/mechanize.rb:443:in
> `fetch_page'
>        from 
>        /usr/local/lib/ruby/gems/1.8/gems/mechanize-0.6.8/lib/mechanize.rb:357:in
> `post_form'
>        from 
>        /usr/local/lib/ruby/gems/1.8/gems/mechanize-0.6.8/lib/mechanize.rb:234:in
> `submit'
>        from (irb):28
>        from :0
> 
> I'd be happy to provide any other information needed to help debug the
> problem.

Can you possibly send a small script that I can run to reproduce the
problem?  I can't tell exactly what the issue is from the stack trace.
Also, setting the logger on mechanize and sending in the logs would help
too.

-- 
Aaron Patterson
http://tenderlovemaking.com/

From mguterl at gmail.com  Thu May 24 09:08:52 2007
From: mguterl at gmail.com (Michael Guterl)
Date: Thu, 24 May 2007 09:08:52 -0400
Subject: [Mechanize-users] Error when posting form
In-Reply-To: <20070523215746.GB10770@hasselhoff.lan>
References: <944a03770705231127y3b786b59n41615c133e65fa02@mail.gmail.com>
	<20070523215746.GB10770@hasselhoff.lan>
Message-ID: <944a03770705240608h565ff9b4nc344baabb4842214@mail.gmail.com>

Hello Aaron,

On 5/23/07, Aaron Patterson <aaron at tenderlovemaking.com> wrote:
>
> Hello Michael,
>
> On Wed, May 23, 2007 at 02:27:32PM -0400, Michael Guterl wrote:
> > Trying to submit a form is resulting in the following error:
> ><SNIP>
> >
> > I'd be happy to provide any other information needed to help debug the
> > problem.
>
> Can you possibly send a small script that I can run to reproduce the
> problem?  I can't tell exactly what the issue is from the stack trace.
> Also, setting the logger on mechanize and sending in the logs would help
> too.


Here's the script with censored URL...  Let me know if you need anything
else.

Thanks again for the great library!

require 'mechanize'
require 'logger'

agent = WWW::Mechanize.new
agent.log = Logger.new(File.new("debug.log", "w+"))
page = agent.get("https://censored.com/dealerApply/")
form = page.forms.first
form.fields.name('clientID').options[1].select
page = agent.submit(form, form.buttons.first)

I, [2007-05-24T09:02:03.617629 #1248]  INFO -- : Net::HTTP::Get:
/dealerApply/
D, [2007-05-24T09:02:03.812380 #1248] DEBUG -- : request-header:
accept-language => en-us,en;q0.5
D, [2007-05-24T09:02:03.812640 #1248] DEBUG -- : request-header: connection
=> keep-alive
D, [2007-05-24T09:02:03.812927 #1248] DEBUG -- : request-header: accept =>
*/*
D, [2007-05-24T09:02:03.813076 #1248] DEBUG -- : request-header:
accept-encoding => gzip,identity
D, [2007-05-24T09:02:03.813224 #1248] DEBUG -- : request-header: user-agent
=> WWW-Mechanize/0.6.8 (http://rubyforge.org/projects/mechanize/)
D, [2007-05-24T09:02:03.813386 #1248] DEBUG -- : request-header:
accept-charset => ISO-8859-1,utf-8;q=0.7,*;q=0.7
D, [2007-05-24T09:02:03.813531 #1248] DEBUG -- : request-header: keep-alive
=> 300
D, [2007-05-24T09:02:03.976138 #1248] DEBUG -- : Read 782 bytes
D, [2007-05-24T09:02:03.977916 #1248] DEBUG -- : Read 1806 bytes
D, [2007-05-24T09:02:03.979310 #1248] DEBUG -- : Read 2830 bytes
D, [2007-05-24T09:02:04.012022 #1248] DEBUG -- : Read 3854 bytes
D, [2007-05-24T09:02:04.013683 #1248] DEBUG -- : Read 4072 bytes
D, [2007-05-24T09:02:04.015736 #1248] DEBUG -- : Read 5091 bytes
D, [2007-05-24T09:02:04.031072 #1248] DEBUG -- : Read 6115 bytes
D, [2007-05-24T09:02:04.032569 #1248] DEBUG -- : Read 7139 bytes
D, [2007-05-24T09:02:04.033803 #1248] DEBUG -- : Read 8144 bytes
D, [2007-05-24T09:02:04.042607 #1248] DEBUG -- : Read 9163 bytes
D, [2007-05-24T09:02:04.047887 #1248] DEBUG -- : Read 10187 bytes
D, [2007-05-24T09:02:04.049675 #1248] DEBUG -- : Read 11211 bytes
D, [2007-05-24T09:02:04.051657 #1248] DEBUG -- : Read 12216 bytes
D, [2007-05-24T09:02:04.054323 #1248] DEBUG -- : Read 13235 bytes
D, [2007-05-24T09:02:04.060651 #1248] DEBUG -- : Read 14259 bytes
D, [2007-05-24T09:02:04.063685 #1248] DEBUG -- : Read 15283 bytes
D, [2007-05-24T09:02:04.065266 #1248] DEBUG -- : Read 16288 bytes
D, [2007-05-24T09:02:04.067075 #1248] DEBUG -- : Read 17307 bytes
D, [2007-05-24T09:02:04.068794 #1248] DEBUG -- : Read 18331 bytes
D, [2007-05-24T09:02:04.070233 #1248] DEBUG -- : Read 18580 bytes
D, [2007-05-24T09:02:04.080093 #1248] DEBUG -- : response-header:
content-type => text/html
D, [2007-05-24T09:02:04.080353 #1248] DEBUG -- : response-header: date =>
Thu, 24 May 2007 13:02:00 GMT
D, [2007-05-24T09:02:04.080460 #1248] DEBUG -- : response-header: server =>
Sun-ONE-Web-Server/6.1
D, [2007-05-24T09:02:04.080546 #1248] DEBUG -- : response-header: set-cookie
=>
JSESSIONID=GVMLvHJPBZpnnGgWtdC4JnDQ7mKd9JvXxHrRYS6qzvwn5J1BKw7g!1550566276;
path=/
D, [2007-05-24T09:02:04.080627 #1248] DEBUG -- : response-header:
transfer-encoding => chunked
D, [2007-05-24T09:02:04.428956 #1248] DEBUG -- : saved cookie:
JSESSIONID=GVMLvHJPBZpnnGgWtdC4JnDQ7mKd9JvXxHrRYS6qzvwn5J1BKw7g!1550566276
I, [2007-05-24T09:02:04.429241 #1248]  INFO -- : status: 200
D, [2007-05-24T09:03:23.612829 #1248] DEBUG -- : query:
"action=start&quick=Y&clientApp=0&clientID=ACE"
I, [2007-05-24T09:03:23.613051 #1248]  INFO -- : Net::HTTP::Post:
/dealerApply/controller.jsp
D, [2007-05-24T09:03:23.613420 #1248] DEBUG -- : using cookie:
JSESSIONID=GVMLvHJPBZpnnGgWtdC4JnDQ7mKd9JvXxHrRYS6qzvwn5J1BKw7g!1550566276
D, [2007-05-24T09:03:23.613737 #1248] DEBUG -- : request-header:
accept-language => en-us,en;q0.5
D, [2007-05-24T09:03:23.613838 #1248] DEBUG -- : request-header: connection
=> keep-alive
D, [2007-05-24T09:03:23.613917 #1248] DEBUG -- : request-header: accept =>
*/*
D, [2007-05-24T09:03:23.613993 #1248] DEBUG -- : request-header:
accept-encoding => gzip,identity
D, [2007-05-24T09:03:23.614070 #1248] DEBUG -- : request-header: user-agent
=> WWW-Mechanize/0.6.8 (http://rubyforge.org/projects/mechanize/)
D, [2007-05-24T09:03:23.614149 #1248] DEBUG -- : request-header:
content-type => application/x-www-form-urlencoded
D, [2007-05-24T09:03:23.614227 #1248] DEBUG -- : request-header: cookie =>
JSESSIONID=GVMLvHJPBZpnnGgWtdC4JnDQ7mKd9JvXxHrRYS6qzvwn5J1BKw7g!1550566276
D, [2007-05-24T09:03:23.614304 #1248] DEBUG -- : request-header: referer =>
https://censored.com/dealerApply/
D, [2007-05-24T09:03:23.614382 #1248] DEBUG -- : request-header:
accept-charset => ISO-8859-1,utf-8;q=0.7,*;q=0.7
D, [2007-05-24T09:03:23.615138 #1248] DEBUG -- : request-header:
content-length => 45
D, [2007-05-24T09:03:23.615277 #1248] DEBUG -- : request-header: keep-alive
=> 300


--
> Aaron Patterson
> http://tenderlovemaking.com/
> _______________________________________________
> Mechanize-users mailing list
> Mechanize-users at rubyforge.org
> http://rubyforge.org/mailman/listinfo/mechanize-users
>

Thanks,
Michael Guterl
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://rubyforge.org/pipermail/mechanize-users/attachments/20070524/a2e3d6e6/attachment.html 

From peter at rubyrailways.com  Mon May 28 03:25:20 2007
From: peter at rubyrailways.com (Peter Szinek)
Date: Mon, 28 May 2007 09:25:20 +0200
Subject: [Mechanize-users] Clicking image maps?
Message-ID: <465A83E0.3000802@rubyrailways.com>

Hello all,

I am trying to click image maps ("area" nodes), and while Perl's 
mechanize treats "a", "area", "frame", "iframe" and "meta" tags as 
links, it seems to me that Ruby's Mechanize thinks only "a" tags can be 
links (at least my little experiment, namely:

page.links.each {|link| puts link.node.name}

on a page with "area" nodes all over tells me).

Is there a way to click "area" nodes? Or at least is there a method of 
Mechanize::Page which returns all the nodes (which would make this 
relatively easy).

2 related questions:

- if I have a HPricot node, which has a 'href' or 'src' attribute, can I 
click it?
- if I have an URI (as a String), can I click it? (or navigate to that URI).

Thanks a lot,
Peter

From aaron at tenderlovemaking.com  Mon May 28 16:12:49 2007
From: aaron at tenderlovemaking.com (Aaron Patterson)
Date: Mon, 28 May 2007 13:12:49 -0700
Subject: [Mechanize-users] Clicking image maps?
In-Reply-To: <465A83E0.3000802@rubyrailways.com>
References: <465A83E0.3000802@rubyrailways.com>
Message-ID: <20070528201249.GA16184@mac-mini.lan>

Hey Peter!

On Mon, May 28, 2007 at 09:25:20AM +0200, Peter Szinek wrote:
> Hello all,
> 
> I am trying to click image maps ("area" nodes), and while Perl's 
> mechanize treats "a", "area", "frame", "iframe" and "meta" tags as 
> links, it seems to me that Ruby's Mechanize thinks only "a" tags can be 
> links (at least my little experiment, namely:

Looks like I've missed area nodes.  Thats a bug, and I'll add it to the
next release.

> 
> page.links.each {|link| puts link.node.name}
> 
> on a page with "area" nodes all over tells me).
> 
> Is there a way to click "area" nodes? Or at least is there a method of 
> Mechanize::Page which returns all the nodes (which would make this 
> relatively easy).

If you find the HPricot node from the page, you should be able to click
it as long as it has an 'href' or a 'src' attribute.

> 
> 2 related questions:
> 
> - if I have a HPricot node, which has a 'href' or 'src' attribute, can I 
> click it?

Yes.  Just pass the node to WWW::Mechanize#click.  It will check for an
href attribute, src attribute, then try calling an href method.  In that
order.

> - if I have an URI (as a String), can I click it? (or navigate to that URI).

Yes.  WWW::Mechanize#get takes a string.

Hope that helps!

--
Aaron Patterson
http://tenderlovemaking.com/

From mguterl at gmail.com  Tue May 29 09:45:19 2007
From: mguterl at gmail.com (Michael Guterl)
Date: Tue, 29 May 2007 09:45:19 -0400
Subject: [Mechanize-users] Error when posting form
In-Reply-To: <944a03770705240608h565ff9b4nc344baabb4842214@mail.gmail.com>
References: <944a03770705231127y3b786b59n41615c133e65fa02@mail.gmail.com>
	<20070523215746.GB10770@hasselhoff.lan>
	<944a03770705240608h565ff9b4nc344baabb4842214@mail.gmail.com>
Message-ID: <944a03770705290645j24a4a8e4i16a457b0eca99ce@mail.gmail.com>

Aaron,

On 5/24/07, Michael Guterl <mguterl at gmail.com> wrote:
>
> Hello Aaron,
>
> On 5/23/07, Aaron Patterson <aaron at tenderlovemaking.com> wrote:
> >
> > Hello Michael,
> >
> > On Wed, May 23, 2007 at 02:27:32PM -0400, Michael Guterl wrote:
> > > Trying to submit a form is resulting in the following error:
> > ><SNIP>
> > >
> > > I'd be happy to provide any other information needed to help debug the
> >
> > > problem.
> >
> > Can you possibly send a small script that I can run to reproduce the
> > problem?  I can't tell exactly what the issue is from the stack trace.
> > Also, setting the logger on mechanize and sending in the logs would help
> >
> > too.
>
>
> Here's the script with censored URL...  Let me know if you need anything
> else.
>
> Thanks again for the great library!
>
> require 'mechanize'
> require 'logger'
>
> agent = WWW::Mechanize.new
> agent.log = Logger.new(File.new("debug.log", "w+"))
> page = agent.get("https://censored.com/dealerApply/")
> form = page.forms.first
> form.fields.name('clientID').options[1].select
> page = agent.submit(form, form.buttons.first)
>
> I, [2007-05-24T09:02:03.617629 #1248]  INFO -- : Net::HTTP::Get:
> /dealerApply/
> D, [2007-05-24T09:02: 03.812380 #1248] DEBUG -- : request-header:
> accept-language => en-us,en;q0.5
> D, [2007-05-24T09:02:03.812640 #1248] DEBUG -- : request-header:
> connection => keep-alive
> D, [2007-05-24T09:02:03.812927 #1248] DEBUG -- : request-header: accept =>
> */*
> D, [2007-05-24T09:02:03.813076 #1248] DEBUG -- : request-header:
> accept-encoding => gzip,identity
> D, [2007-05-24T09:02:03.813224 #1248] DEBUG -- : request-header:
> user-agent => WWW-Mechanize/0.6.8 (http://rubyforge.org/projects/mechanize/
> )
> D, [2007-05-24T09:02:03.813386 #1248] DEBUG -- : request-header:
> accept-charset => ISO-8859-1,utf-8;q=0.7,*;q=0.7
> D, [2007-05-24T09:02:03.813531 #1248] DEBUG -- : request-header:
> keep-alive => 300
> D, [2007-05-24T09:02:03.976138 #1248] DEBUG -- : Read 782 bytes
> D, [2007-05-24T09:02:03.977916 #1248] DEBUG -- : Read 1806 bytes
> D, [2007-05-24T09:02:03.979310 #1248] DEBUG -- : Read 2830 bytes
> D, [2007-05-24T09:02: 04.012022 #1248] DEBUG -- : Read 3854 bytes
> D, [2007-05-24T09:02:04.013683 #1248] DEBUG -- : Read 4072 bytes
> D, [2007-05-24T09:02:04.015736 #1248] DEBUG -- : Read 5091 bytes
> D, [2007-05-24T09:02:04.031072 #1248] DEBUG -- : Read 6115 bytes
> D, [2007-05-24T09:02:04.032569 #1248] DEBUG -- : Read 7139 bytes
> D, [2007-05-24T09:02:04.033803 #1248] DEBUG -- : Read 8144 bytes
> D, [2007-05-24T09:02:04.042607 #1248] DEBUG -- : Read 9163 bytes
> D, [2007-05-24T09:02: 04.047887 #1248] DEBUG -- : Read 10187 bytes
> D, [2007-05-24T09:02:04.049675 #1248] DEBUG -- : Read 11211 bytes
> D, [2007-05-24T09:02:04.051657 #1248] DEBUG -- : Read 12216 bytes
> D, [2007-05-24T09:02:04.054323 #1248] DEBUG -- : Read 13235 bytes
> D, [2007-05-24T09:02:04.060651 #1248] DEBUG -- : Read 14259 bytes
> D, [2007-05-24T09:02:04.063685 #1248] DEBUG -- : Read 15283 bytes
> D, [2007-05-24T09:02:04.065266 #1248] DEBUG -- : Read 16288 bytes
> D, [2007-05-24T09:02: 04.067075 #1248] DEBUG -- : Read 17307 bytes
> D, [2007-05-24T09:02:04.068794 #1248] DEBUG -- : Read 18331 bytes
> D, [2007-05-24T09:02:04.070233 #1248] DEBUG -- : Read 18580 bytes
> D, [2007-05-24T09:02:04.080093 #1248] DEBUG -- : response-header:
> content-type => text/html
> D, [2007-05-24T09:02:04.080353 #1248] DEBUG -- : response-header: date =>
> Thu, 24 May 2007 13:02:00 GMT
> D, [2007-05-24T09:02:04.080460 #1248] DEBUG -- : response-header: server
> => Sun-ONE-Web-Server/6.1
> D, [2007-05-24T09:02: 04.080546 #1248] DEBUG -- : response-header:
> set-cookie =>
> JSESSIONID=GVMLvHJPBZpnnGgWtdC4JnDQ7mKd9JvXxHrRYS6qzvwn5J1BKw7g!1550566276;
> path=/
> D, [2007-05-24T09:02:04.080627 #1248] DEBUG -- : response-header:
> transfer-encoding => chunked
> D, [2007-05-24T09:02:04.428956 #1248] DEBUG -- : saved cookie:
> JSESSIONID=GVMLvHJPBZpnnGgWtdC4JnDQ7mKd9JvXxHrRYS6qzvwn5J1BKw7g!1550566276
> I, [2007-05-24T09:02:04.429241 #1248]  INFO -- : status: 200
> D, [2007-05-24T09:03: 23.612829 #1248] DEBUG -- : query:
> "action=start&quick=Y&clientApp=0&clientID=ACE"
> I, [2007-05-24T09:03:23.613051 #1248]  INFO -- : Net::HTTP::Post:
> /dealerApply/controller.jsp
> D, [2007-05-24T09:03: 23.613420 #1248] DEBUG -- : using cookie:
> JSESSIONID=GVMLvHJPBZpnnGgWtdC4JnDQ7mKd9JvXxHrRYS6qzvwn5J1BKw7g!1550566276
> D, [2007-05-24T09:03:23.613737 #1248] DEBUG -- : request-header:
> accept-language => en-us,en;q0.5
> D, [2007-05-24T09:03:23.613838 #1248] DEBUG -- : request-header:
> connection => keep-alive
> D, [2007-05-24T09:03:23.613917 #1248] DEBUG -- : request-header: accept =>
> */*
> D, [2007-05-24T09:03:23.613993 #1248] DEBUG -- : request-header:
> accept-encoding => gzip,identity
> D, [2007-05-24T09:03:23.614070 #1248] DEBUG -- : request-header:
> user-agent => WWW-Mechanize/0.6.8 (
> http://rubyforge.org/projects/mechanize/)
> D, [2007-05-24T09:03: 23.614149 #1248] DEBUG -- : request-header:
> content-type => application/x-www-form-urlencoded
> D, [2007-05-24T09:03:23.614227 #1248] DEBUG -- : request-header: cookie =>
> JSESSIONID=GVMLvHJPBZpnnGgWtdC4JnDQ7mKd9JvXxHrRYS6qzvwn5J1BKw7g!1550566276
> D, [2007-05-24T09:03:23.614304 #1248] DEBUG -- : request-header: referer
> => https://censored.com/dealerApply/
> D, [2007-05-24T09:03:23.614382 #1248] DEBUG -- : request-header:
> accept-charset => ISO-8859-1,utf-8;q= 0.7,*;q=0.7
> D, [2007-05-24T09:03:23.615138 #1248] DEBUG -- : request-header:
> content-length => 45
> D, [2007-05-24T09:03:23.615277 #1248] DEBUG -- : request-header:
> keep-alive => 300
>
>
> --
> > Aaron Patterson
> > http://tenderlovemaking.com/
> > _______________________________________________
> > Mechanize-users mailing list
> > Mechanize-users at rubyforge.org
> > http://rubyforge.org/mailman/listinfo/mechanize-users
> >
>
> Thanks,
> Michael Guterl
>
>
Do I need to provide you with any more information?

Thanks,
Michael Guterl
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://rubyforge.org/pipermail/mechanize-users/attachments/20070529/a8a23346/attachment-0001.html 

From peter at rubyrailways.com  Wed May 30 03:33:29 2007
From: peter at rubyrailways.com (Peter Szinek)
Date: Wed, 30 May 2007 09:33:29 +0200
Subject: [Mechanize-users] Clicking image maps?
In-Reply-To: <20070528201249.GA16184@mac-mini.lan>
References: <465A83E0.3000802@rubyrailways.com>
	<20070528201249.GA16184@mac-mini.lan>
Message-ID: <465D28C9.2040605@rubyrailways.com>

Aaron,

> Hope that helps!

Definitely! Thanks a lot for your great support!

Cheers,
Peter

From aaron at tenderlovemaking.com  Wed May 30 13:17:19 2007
From: aaron at tenderlovemaking.com (Aaron Patterson)
Date: Wed, 30 May 2007 10:17:19 -0700
Subject: [Mechanize-users] Error when posting form
In-Reply-To: <944a03770705290645j24a4a8e4i16a457b0eca99ce@mail.gmail.com>
References: <944a03770705231127y3b786b59n41615c133e65fa02@mail.gmail.com>
	<20070523215746.GB10770@hasselhoff.lan>
	<944a03770705240608h565ff9b4nc344baabb4842214@mail.gmail.com>
	<944a03770705290645j24a4a8e4i16a457b0eca99ce@mail.gmail.com>
Message-ID: <20070530171718.GA24385@mac-mini.lan>

On Tue, May 29, 2007 at 09:45:19AM -0400, Michael Guterl wrote:
> Aaron,
> 

[snip]

> >
> >agent = WWW::Mechanize.new
> >agent.log = Logger.new(File.new("debug.log", "w+"))
> >page = agent.get("https://censored.com/dealerApply/")
> >form = page.forms.first
> >form.fields.name('clientID').options[1].select
> >page = agent.submit(form, form.buttons.first)

[snip]

> Do I need to provide you with any more information?

Hi Michael,

I don't see anything wrong with your script, and I can't tell what is
causing the problem from the logs.  Unfortunately in this case it looks
like I would have to actually execute the script.

Is there any way you can reproduce the problem using a website where you
don't mind sharing the URL?  I would really like to help, but this just
seems like one of those times I'll need a script I can execute.

-- 
Aaron Patterson
http://tenderlovemaking.com/