From drbrain at segment7.net  Wed May  9 20:48:18 2012
From: drbrain at segment7.net (Eric Hodel)
Date: Wed, 9 May 2012 13:48:18 -0700
Subject: [Mechanize-users] [ANN] mechanize 2.5 Released
Message-ID: <2BD1F3A4-BA73-45FC-9B7E-6C465E232270@segment7.net>

mechanize version 2.5 has been released!

* http://mechanize.rubyforge.org
* https://github.com/tenderlove/mechanize

The Mechanize library is used for automating interaction with websites.
Mechanize automatically stores and sends cookies, follows redirects,
and can follow links and submit forms.  Form fields can be populated and
submitted.  Mechanize also keeps track of the sites that you have visited as
a history.

=== Changes

* Minor enhancement
  * Added Mechanize#ignore_bad_chunking for working around servers that don't
    terminate chunked transfer-encoding properly.  Enabling this may cause
    data loss.  Issue #116
  * Removed content-type check from Mechanize::Page allowing forced parsing
    of incorrect or missing content-types.  Issue #221 by GarthSnyder
* Bug fixes
  * Fixed typos in EXAMPLES and GUIDES.  Pull Request #213 by Erkan Yilmaz.
  * Fixed handling of a quoted content-disposition size.  Pull Request #220 by
    Jason Rust
  * Mechanize now ignores a missing gzip footer like browsers do.  Issue #224
    by afhbl
  * Mechanize handles saving of files with the same name better now.  Pull
    Request #223 by Godfrey Chan, Issue #219 by Jon Hart
  * Mechanize now sends headers across redirects.  Issue #215 by Chris Gahan
  * Mechanize now raises Mechanize::ResponseReadError when the server does not
    terminate chunked transfer-encoding properly.  Issue #116
  * Mechanize no longer raises an exception when multiple identical
    radiobuttons are checked.  Issue #214 by Matthias Guenther
  * Fixed documentation for pre_connect_hooks and post_connect_hooks.  Issue
    #226 by Robert Poor
  * Worked around ruby 1.8 run with -Ku and ISO-8859-1 encoded characters in
    URIs.  Issue #228 by Stanislav O.Pogrebnyak


From drbrain at segment7.net  Wed May 16 02:30:11 2012
From: drbrain at segment7.net (Eric Hodel)
Date: Tue, 15 May 2012 19:30:11 -0700
Subject: [Mechanize-users] [ANN] mechanize 2.5.1 Released
Message-ID: <A4577C7A-0798-4B23-A99E-55C5DFBF3F9F@segment7.net>

mechanize version 2.5.1 has been released!

* <http://mechanize.rubyforge.org>
* <https://github.com/tenderlove/mechanize>

The Mechanize library is used for automating interaction with websites.
Mechanize automatically stores and sends cookies, follows redirects,
and can follow links and submit forms.  Form fields can be populated and
submitted.  Mechanize also keeps track of the sites that you have visited as
a history.

Changes:

* Bug fix
  * Mechanize no longer copies POST requests during a redirect which was
    introduced by #215.  Pull request #229 by Godfrey Chan.


From kevin at proofimaging.com  Wed May 16 04:39:53 2012
From: kevin at proofimaging.com (Kevin Kornemann)
Date: Wed, 16 May 2012 00:39:53 -0400
Subject: [Mechanize-users] [ANN] mechanize 2.5.1 Released
In-Reply-To: <A4577C7A-0798-4B23-A99E-55C5DFBF3F9F@segment7.net>
References: <A4577C7A-0798-4B23-A99E-55C5DFBF3F9F@segment7.net>
Message-ID: <CA+TZ5hBfD_U0zBnz0pJF695teigMaFyvGJx-WtScHgNyjGmA-g@mail.gmail.com>

Does this handle js better than older versions?

On Tue, May 15, 2012 at 10:30 PM, Eric Hodel <drbrain at segment7.net> wrote:

> mechanize version 2.5.1 has been released!
>
> * <http://mechanize.rubyforge.org>
> * <https://github.com/tenderlove/mechanize>
>
> The Mechanize library is used for automating interaction with websites.
> Mechanize automatically stores and sends cookies, follows redirects,
> and can follow links and submit forms.  Form fields can be populated and
> submitted.  Mechanize also keeps track of the sites that you have visited
> as
> a history.
>
> Changes:
>
> * Bug fix
>  * Mechanize no longer copies POST requests during a redirect which was
>    introduced by #215.  Pull request #229 by Godfrey Chan.
>
> _______________________________________________
> Mechanize-users mailing list
> Mechanize-users at rubyforge.org
> http://rubyforge.org/mailman/listinfo/mechanize-users
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://rubyforge.org/pipermail/mechanize-users/attachments/20120516/7c9f5f78/attachment.html>

From drbrain at segment7.net  Wed May 16 18:26:26 2012
From: drbrain at segment7.net (Eric Hodel)
Date: Wed, 16 May 2012 11:26:26 -0700
Subject: [Mechanize-users] [ANN] mechanize 2.5.1 Released
In-Reply-To: <CA+TZ5hBfD_U0zBnz0pJF695teigMaFyvGJx-WtScHgNyjGmA-g@mail.gmail.com>
References: <A4577C7A-0798-4B23-A99E-55C5DFBF3F9F@segment7.net>
	<CA+TZ5hBfD_U0zBnz0pJF695teigMaFyvGJx-WtScHgNyjGmA-g@mail.gmail.com>
Message-ID: <60F1B01D-C1F5-45F8-8C33-2C2F15BF91BE@segment7.net>

On May 15, 2012, at 21:39, Kevin Kornemann wrote:
> Does this handle js better than older versions?

It handles JavaScript exactly the same as every other version, you can download them and display them, but not execute them.

Execution of JavaScript requires a JavaScript interpreter and DOM implementation atop Nokogiri, both of which are beyond the scope of mechanize.

From kevin at proofimaging.com  Wed May 16 18:35:59 2012
From: kevin at proofimaging.com (Kevin Kornemann)
Date: Wed, 16 May 2012 14:35:59 -0400
Subject: [Mechanize-users] [ANN] mechanize 2.5.1 Released
In-Reply-To: <60F1B01D-C1F5-45F8-8C33-2C2F15BF91BE@segment7.net>
References: <A4577C7A-0798-4B23-A99E-55C5DFBF3F9F@segment7.net>
	<CA+TZ5hBfD_U0zBnz0pJF695teigMaFyvGJx-WtScHgNyjGmA-g@mail.gmail.com>
	<60F1B01D-C1F5-45F8-8C33-2C2F15BF91BE@segment7.net>
Message-ID: <CA+TZ5hDNA79SHCO3Jvv6bqNdUh0LGyWWnKZuStBK6F8nr0wPUg@mail.gmail.com>

cool, thanks! I'm trying to scrape a page that requires login, and then
clicking on form check boxes and then hitting search. I managed to get this
far with mechanize but the search results I get are garbage (not what I get
when I do the same with a browser), so I was grasping at straws that it was
a js script not being executed problem, but I really don't know. Looking
into other options but realy nothing I found was as easy or nice to use as
mechanize.

If it's not too much trouble, any hints are appreciated... to re-woord my
problem, search results are not as expected, even wrong page is being
returned from what I can tell. This is my script:

require 'nokogiri'
require 'mechanize'

agent = Mechanize.new

agent.follow_meta_refresh = true
agent.user_agent_alias = 'Mac Safari'

page = agent.get('http://connect.raps.org/RAPS/Directory/AdvancedSearch/')

login_form = page.form()

login_form.LoginTextBox = 'alexa coppola'
login_form.PasswordTextBox = 'goodfriend123'

page = agent.submit(login_form, login_form.buttons.first)

agent.follow_meta_refresh = false

search_form = page.form()

search_form.checkbox_with(:name =>
'ctl00$MainCopy$ctl08$DemographicsList$0').check

page = agent.submit(search_form, search_form.buttons.first)


# returns different page than expected, not search results.

On Wed, May 16, 2012 at 2:26 PM, Eric Hodel <drbrain at segment7.net> wrote:

> On May 15, 2012, at 21:39, Kevin Kornemann wrote:
> > Does this handle js better than older versions?
>
> It handles JavaScript exactly the same as every other version, you can
> download them and display them, but not execute them.
>
> Execution of JavaScript requires a JavaScript interpreter and DOM
> implementation atop Nokogiri, both of which are beyond the scope of
> mechanize.
> _______________________________________________
> Mechanize-users mailing list
> Mechanize-users at rubyforge.org
> http://rubyforge.org/mailman/listinfo/mechanize-users
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://rubyforge.org/pipermail/mechanize-users/attachments/20120516/a3df2d5f/attachment.html>

From drbrain at segment7.net  Wed May 16 20:25:33 2012
From: drbrain at segment7.net (Eric Hodel)
Date: Wed, 16 May 2012 13:25:33 -0700
Subject: [Mechanize-users] [ANN] mechanize 2.5.1 Released
In-Reply-To: <CA+TZ5hDNA79SHCO3Jvv6bqNdUh0LGyWWnKZuStBK6F8nr0wPUg@mail.gmail.com>
References: <A4577C7A-0798-4B23-A99E-55C5DFBF3F9F@segment7.net>
	<CA+TZ5hBfD_U0zBnz0pJF695teigMaFyvGJx-WtScHgNyjGmA-g@mail.gmail.com>
	<60F1B01D-C1F5-45F8-8C33-2C2F15BF91BE@segment7.net>
	<CA+TZ5hDNA79SHCO3Jvv6bqNdUh0LGyWWnKZuStBK6F8nr0wPUg@mail.gmail.com>
Message-ID: <2549C44D-03C7-418F-B558-FE6F6BD4A38D@segment7.net>

On May 16, 2012, at 11:35, Kevin Kornemann wrote:
> cool, thanks! I'm trying to scrape a page that requires login, and then clicking on form check boxes and then hitting search. I managed to get this far with mechanize but the search results I get are garbage (not what I get when I do the same with a browser), so I was grasping at straws that it was a js script not being executed problem, but I really don't know. Looking into other options but realy nothing I found was as easy or nice to use as mechanize.
> 
> If it's not too much trouble, any hints are appreciated... to re-woord my problem, search results are not as expected, even wrong page is being returned from what I can tell. This is my script:
> 
> require 'nokogiri'
> require 'mechanize'
> 
> agent = Mechanize.new
> 
> agent.follow_meta_refresh = true
> agent.user_agent_alias = 'Mac Safari'
> 
> page = agent.get('http://connect.raps.org/RAPS/Directory/AdvancedSearch/')
> 
> login_form = page.form()
> 
> login_form.LoginTextBox = 'alexa coppola'
> login_form.PasswordTextBox = 'goodfriend123'
> 
> page = agent.submit(login_form, login_form.buttons.first)
> 
> agent.follow_meta_refresh = false
> 
> search_form = page.form()
> 
> search_form.checkbox_with(:name => 'ctl00$MainCopy$ctl08$DemographicsList$0').check
> 
> page = agent.submit(search_form, search_form.buttons.first)
> 
> 
> # returns different page than expected, not search results.

You're going to need to emulate what the javascript does when you click on the request page.  You can use browser debugging tools to compare what query a browser makes with what query mechanize makes (through use of the debugger).

PS:  This is a public list so you will want to change the password.

From kevin at proofimaging.com  Wed May 16 20:38:03 2012
From: kevin at proofimaging.com (Kevin Kornemann)
Date: Wed, 16 May 2012 16:38:03 -0400
Subject: [Mechanize-users] [ANN] mechanize 2.5.1 Released
In-Reply-To: <2549C44D-03C7-418F-B558-FE6F6BD4A38D@segment7.net>
References: <A4577C7A-0798-4B23-A99E-55C5DFBF3F9F@segment7.net>
	<CA+TZ5hBfD_U0zBnz0pJF695teigMaFyvGJx-WtScHgNyjGmA-g@mail.gmail.com>
	<60F1B01D-C1F5-45F8-8C33-2C2F15BF91BE@segment7.net>
	<CA+TZ5hDNA79SHCO3Jvv6bqNdUh0LGyWWnKZuStBK6F8nr0wPUg@mail.gmail.com>
	<2549C44D-03C7-418F-B558-FE6F6BD4A38D@segment7.net>
Message-ID: <CA+TZ5hA5=Tcx4Gk2T1WDMcjPsAEUuCKFMQbiy7=5rkxAD6676g@mail.gmail.com>

shit, didn't realize this was public. Can I delete this post?

On Wed, May 16, 2012 at 4:25 PM, Eric Hodel <drbrain at segment7.net> wrote:

> On May 16, 2012, at 11:35, Kevin Kornemann wrote:
> > cool, thanks! I'm trying to scrape a page that requires login, and then
> clicking on form check boxes and then hitting search. I managed to get this
> far with mechanize but the search results I get are garbage (not what I get
> when I do the same with a browser), so I was grasping at straws that it was
> a js script not being executed problem, but I really don't know. Looking
> into other options but realy nothing I found was as easy or nice to use as
> mechanize.
> >
> > If it's not too much trouble, any hints are appreciated... to re-woord
> my problem, search results are not as expected, even wrong page is being
> returned from what I can tell. This is my script:
> >
> > require 'nokogiri'
> > require 'mechanize'
> >
> > agent = Mechanize.new
> >
> > agent.follow_meta_refresh = true
> > agent.user_agent_alias = 'Mac Safari'
> >
> > page = agent.get('http://connect.raps.org/RAPS/Directory/AdvancedSearch/
> ')
> >
> > login_form = page.form()
> >
> > login_form.LoginTextBox = 'alexa coppola'
> > login_form.PasswordTextBox = 'goodfriend123'
> >
> > page = agent.submit(login_form, login_form.buttons.first)
> >
> > agent.follow_meta_refresh = false
> >
> > search_form = page.form()
> >
> > search_form.checkbox_with(:name =>
> 'ctl00$MainCopy$ctl08$DemographicsList$0').check
> >
> > page = agent.submit(search_form, search_form.buttons.first)
> >
> >
> > # returns different page than expected, not search results.
>
> You're going to need to emulate what the javascript does when you click on
> the request page.  You can use browser debugging tools to compare what
> query a browser makes with what query mechanize makes (through use of the
> debugger).
>
> PS:  This is a public list so you will want to change the password.
> _______________________________________________
> Mechanize-users mailing list
> Mechanize-users at rubyforge.org
> http://rubyforge.org/mailman/listinfo/mechanize-users
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://rubyforge.org/pipermail/mechanize-users/attachments/20120516/b0595c94/attachment.html>

From sirbeep at gmail.com  Wed May 16 20:47:28 2012
From: sirbeep at gmail.com (Brian Kennedy)
Date: Wed, 16 May 2012 16:47:28 -0400
Subject: [Mechanize-users] [ANN] mechanize 2.5.1 Released
In-Reply-To: <CA+TZ5hA5=Tcx4Gk2T1WDMcjPsAEUuCKFMQbiy7=5rkxAD6676g@mail.gmail.com>
References: <A4577C7A-0798-4B23-A99E-55C5DFBF3F9F@segment7.net>
	<CA+TZ5hBfD_U0zBnz0pJF695teigMaFyvGJx-WtScHgNyjGmA-g@mail.gmail.com>
	<60F1B01D-C1F5-45F8-8C33-2C2F15BF91BE@segment7.net>
	<CA+TZ5hDNA79SHCO3Jvv6bqNdUh0LGyWWnKZuStBK6F8nr0wPUg@mail.gmail.com>
	<2549C44D-03C7-418F-B558-FE6F6BD4A38D@segment7.net>
	<CA+TZ5hA5=Tcx4Gk2T1WDMcjPsAEUuCKFMQbiy7=5rkxAD6676g@mail.gmail.com>
Message-ID: <CACYs=59XQw0GuHR-xtGCR02PGSFvt36ztHycQT90uRb7GuZC1Q@mail.gmail.com>

It's in my email inbox,  so nope.  :)

Cat's already shredded the bag and is working on the curtains.  Change your
password quick, you don't know who's on the list that may have taken it and
is running with it already.

On Wed, May 16, 2012 at 4:38 PM, Kevin Kornemann <kevin at proofimaging.com>wrote:

> shit, didn't realize this was public. Can I delete this post?
>
>
> On Wed, May 16, 2012 at 4:25 PM, Eric Hodel <drbrain at segment7.net> wrote:
>
>> On May 16, 2012, at 11:35, Kevin Kornemann wrote:
>> > cool, thanks! I'm trying to scrape a page that requires login, and then
>> clicking on form check boxes and then hitting search. I managed to get this
>> far with mechanize but the search results I get are garbage (not what I get
>> when I do the same with a browser), so I was grasping at straws that it was
>> a js script not being executed problem, but I really don't know. Looking
>> into other options but realy nothing I found was as easy or nice to use as
>> mechanize.
>> >
>> > If it's not too much trouble, any hints are appreciated... to re-woord
>> my problem, search results are not as expected, even wrong page is being
>> returned from what I can tell. This is my script:
>> >
>> > require 'nokogiri'
>> > require 'mechanize'
>> >
>> > agent = Mechanize.new
>> >
>> > agent.follow_meta_refresh = true
>> > agent.user_agent_alias = 'Mac Safari'
>> >
>> > page = agent.get('
>> http://connect.raps.org/RAPS/Directory/AdvancedSearch/')
>> >
>> > login_form = page.form()
>> >
>> > login_form.LoginTextBox = 'alexa coppola'
>> > login_form.PasswordTextBox = 'goodfriend123'
>> >
>> > page = agent.submit(login_form, login_form.buttons.first)
>> >
>> > agent.follow_meta_refresh = false
>> >
>> > search_form = page.form()
>> >
>> > search_form.checkbox_with(:name =>
>> 'ctl00$MainCopy$ctl08$DemographicsList$0').check
>> >
>> > page = agent.submit(search_form, search_form.buttons.first)
>> >
>> >
>> > # returns different page than expected, not search results.
>>
>> You're going to need to emulate what the javascript does when you click
>> on the request page.  You can use browser debugging tools to compare what
>> query a browser makes with what query mechanize makes (through use of the
>> debugger).
>>
>> PS:  This is a public list so you will want to change the password.
>> _______________________________________________
>> Mechanize-users mailing list
>> Mechanize-users at rubyforge.org
>> http://rubyforge.org/mailman/listinfo/mechanize-users
>>
>
>
> _______________________________________________
> Mechanize-users mailing list
> Mechanize-users at rubyforge.org
> http://rubyforge.org/mailman/listinfo/mechanize-users
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://rubyforge.org/pipermail/mechanize-users/attachments/20120516/98dff766/attachment.html>

From kevin at proofimaging.com  Wed May 16 20:50:55 2012
From: kevin at proofimaging.com (Kevin Kornemann)
Date: Wed, 16 May 2012 16:50:55 -0400
Subject: [Mechanize-users] [ANN] mechanize 2.5.1 Released
In-Reply-To: <CACYs=59XQw0GuHR-xtGCR02PGSFvt36ztHycQT90uRb7GuZC1Q@mail.gmail.com>
References: <A4577C7A-0798-4B23-A99E-55C5DFBF3F9F@segment7.net>
	<CA+TZ5hBfD_U0zBnz0pJF695teigMaFyvGJx-WtScHgNyjGmA-g@mail.gmail.com>
	<60F1B01D-C1F5-45F8-8C33-2C2F15BF91BE@segment7.net>
	<CA+TZ5hDNA79SHCO3Jvv6bqNdUh0LGyWWnKZuStBK6F8nr0wPUg@mail.gmail.com>
	<2549C44D-03C7-418F-B558-FE6F6BD4A38D@segment7.net>
	<CA+TZ5hA5=Tcx4Gk2T1WDMcjPsAEUuCKFMQbiy7=5rkxAD6676g@mail.gmail.com>
	<CACYs=59XQw0GuHR-xtGCR02PGSFvt36ztHycQT90uRb7GuZC1Q@mail.gmail.com>
Message-ID: <CA+TZ5hDKS5NYg0JsBSwtTkmYrm7FwGRUOudk+myNw+AMK_6z0w@mail.gmail.com>

changed, but didn't get the rest.

On Wed, May 16, 2012 at 4:47 PM, Brian Kennedy <sirbeep at gmail.com> wrote:

> It's in my email inbox,  so nope.  :)
>
> Cat's already shredded the bag and is working on the curtains.  Change
> your password quick, you don't know who's on the list that may have taken
> it and is running with it already.
>
>
> On Wed, May 16, 2012 at 4:38 PM, Kevin Kornemann <kevin at proofimaging.com>wrote:
>
>> shit, didn't realize this was public. Can I delete this post?
>>
>>
>> On Wed, May 16, 2012 at 4:25 PM, Eric Hodel <drbrain at segment7.net> wrote:
>>
>>> On May 16, 2012, at 11:35, Kevin Kornemann wrote:
>>> > cool, thanks! I'm trying to scrape a page that requires login, and
>>> then clicking on form check boxes and then hitting search. I managed to get
>>> this far with mechanize but the search results I get are garbage (not what
>>> I get when I do the same with a browser), so I was grasping at straws that
>>> it was a js script not being executed problem, but I really don't know.
>>> Looking into other options but realy nothing I found was as easy or nice to
>>> use as mechanize.
>>> >
>>> > If it's not too much trouble, any hints are appreciated... to re-woord
>>> my problem, search results are not as expected, even wrong page is being
>>> returned from what I can tell. This is my script:
>>> >
>>> > require 'nokogiri'
>>> > require 'mechanize'
>>> >
>>> > agent = Mechanize.new
>>> >
>>> > agent.follow_meta_refresh = true
>>> > agent.user_agent_alias = 'Mac Safari'
>>> >
>>> > page = agent.get('
>>> http://connect.raps.org/RAPS/Directory/AdvancedSearch/')
>>> >
>>> > login_form = page.form()
>>> >
>>> > login_form.LoginTextBox = 'alexa coppola'
>>> > login_form.PasswordTextBox = 'goodfriend123'
>>> >
>>> > page = agent.submit(login_form, login_form.buttons.first)
>>> >
>>> > agent.follow_meta_refresh = false
>>> >
>>> > search_form = page.form()
>>> >
>>> > search_form.checkbox_with(:name =>
>>> 'ctl00$MainCopy$ctl08$DemographicsList$0').check
>>> >
>>> > page = agent.submit(search_form, search_form.buttons.first)
>>> >
>>> >
>>> > # returns different page than expected, not search results.
>>>
>>> You're going to need to emulate what the javascript does when you click
>>> on the request page.  You can use browser debugging tools to compare what
>>> query a browser makes with what query mechanize makes (through use of the
>>> debugger).
>>>
>>> PS:  This is a public list so you will want to change the password.
>>> _______________________________________________
>>> Mechanize-users mailing list
>>> Mechanize-users at rubyforge.org
>>> http://rubyforge.org/mailman/listinfo/mechanize-users
>>>
>>
>>
>> _______________________________________________
>> Mechanize-users mailing list
>> Mechanize-users at rubyforge.org
>> http://rubyforge.org/mailman/listinfo/mechanize-users
>>
>
>
> _______________________________________________
> Mechanize-users mailing list
> Mechanize-users at rubyforge.org
> http://rubyforge.org/mailman/listinfo/mechanize-users
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://rubyforge.org/pipermail/mechanize-users/attachments/20120516/b7d062c8/attachment.html>

From kevin at proofimaging.com  Wed May 16 20:54:58 2012
From: kevin at proofimaging.com (Kevin Kornemann)
Date: Wed, 16 May 2012 16:54:58 -0400
Subject: [Mechanize-users] [ANN] mechanize 2.5.1 Released
In-Reply-To: <CA+TZ5hDKS5NYg0JsBSwtTkmYrm7FwGRUOudk+myNw+AMK_6z0w@mail.gmail.com>
References: <A4577C7A-0798-4B23-A99E-55C5DFBF3F9F@segment7.net>
	<CA+TZ5hBfD_U0zBnz0pJF695teigMaFyvGJx-WtScHgNyjGmA-g@mail.gmail.com>
	<60F1B01D-C1F5-45F8-8C33-2C2F15BF91BE@segment7.net>
	<CA+TZ5hDNA79SHCO3Jvv6bqNdUh0LGyWWnKZuStBK6F8nr0wPUg@mail.gmail.com>
	<2549C44D-03C7-418F-B558-FE6F6BD4A38D@segment7.net>
	<CA+TZ5hA5=Tcx4Gk2T1WDMcjPsAEUuCKFMQbiy7=5rkxAD6676g@mail.gmail.com>
	<CACYs=59XQw0GuHR-xtGCR02PGSFvt36ztHycQT90uRb7GuZC1Q@mail.gmail.com>
	<CA+TZ5hDKS5NYg0JsBSwtTkmYrm7FwGRUOudk+myNw+AMK_6z0w@mail.gmail.com>
Message-ID: <CA+TZ5hCUiVTzQT_PLPjPA9gF0KmbOc=MXnRipqG_nAwt3sAGHQ@mail.gmail.com>

If possible, can you delete the email in your inbox? I mean surely a posted
can be deleted somehow?

On Wed, May 16, 2012 at 4:50 PM, Kevin Kornemann <kevin at proofimaging.com>wrote:

> changed, but didn't get the rest.
>
>
> On Wed, May 16, 2012 at 4:47 PM, Brian Kennedy <sirbeep at gmail.com> wrote:
>
>> It's in my email inbox,  so nope.  :)
>>
>> Cat's already shredded the bag and is working on the curtains.  Change
>> your password quick, you don't know who's on the list that may have taken
>> it and is running with it already.
>>
>>
>> On Wed, May 16, 2012 at 4:38 PM, Kevin Kornemann <kevin at proofimaging.com>wrote:
>>
>>> shit, didn't realize this was public. Can I delete this post?
>>>
>>>
>>> On Wed, May 16, 2012 at 4:25 PM, Eric Hodel <drbrain at segment7.net>wrote:
>>>
>>>> On May 16, 2012, at 11:35, Kevin Kornemann wrote:
>>>> > cool, thanks! I'm trying to scrape a page that requires login, and
>>>> then clicking on form check boxes and then hitting search. I managed to get
>>>> this far with mechanize but the search results I get are garbage (not what
>>>> I get when I do the same with a browser), so I was grasping at straws that
>>>> it was a js script not being executed problem, but I really don't know.
>>>> Looking into other options but realy nothing I found was as easy or nice to
>>>> use as mechanize.
>>>> >
>>>> > If it's not too much trouble, any hints are appreciated... to
>>>> re-woord my problem, search results are not as expected, even wrong page is
>>>> being returned from what I can tell. This is my script:
>>>> >
>>>> > require 'nokogiri'
>>>> > require 'mechanize'
>>>> >
>>>> > agent = Mechanize.new
>>>> >
>>>> > agent.follow_meta_refresh = true
>>>> > agent.user_agent_alias = 'Mac Safari'
>>>> >
>>>> > page = agent.get('
>>>> http://connect.raps.org/RAPS/Directory/AdvancedSearch/')
>>>> >
>>>> > login_form = page.form()
>>>> >
>>>> > login_form.LoginTextBox = 'alexa coppola'
>>>> > login_form.PasswordTextBox = 'goodfriend123'
>>>> >
>>>> > page = agent.submit(login_form, login_form.buttons.first)
>>>> >
>>>> > agent.follow_meta_refresh = false
>>>> >
>>>> > search_form = page.form()
>>>> >
>>>> > search_form.checkbox_with(:name =>
>>>> 'ctl00$MainCopy$ctl08$DemographicsList$0').check
>>>> >
>>>> > page = agent.submit(search_form, search_form.buttons.first)
>>>> >
>>>> >
>>>> > # returns different page than expected, not search results.
>>>>
>>>> You're going to need to emulate what the javascript does when you click
>>>> on the request page.  You can use browser debugging tools to compare what
>>>> query a browser makes with what query mechanize makes (through use of the
>>>> debugger).
>>>>
>>>> PS:  This is a public list so you will want to change the password.
>>>> _______________________________________________
>>>> Mechanize-users mailing list
>>>> Mechanize-users at rubyforge.org
>>>> http://rubyforge.org/mailman/listinfo/mechanize-users
>>>>
>>>
>>>
>>> _______________________________________________
>>> Mechanize-users mailing list
>>> Mechanize-users at rubyforge.org
>>> http://rubyforge.org/mailman/listinfo/mechanize-users
>>>
>>
>>
>> _______________________________________________
>> Mechanize-users mailing list
>> Mechanize-users at rubyforge.org
>> http://rubyforge.org/mailman/listinfo/mechanize-users
>>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://rubyforge.org/pipermail/mechanize-users/attachments/20120516/3bd7d995/attachment-0001.html>

From sirbeep at gmail.com  Wed May 16 21:00:41 2012
From: sirbeep at gmail.com (Brian Kennedy)
Date: Wed, 16 May 2012 17:00:41 -0400
Subject: [Mechanize-users] [ANN] mechanize 2.5.1 Released
In-Reply-To: <CA+TZ5hCUiVTzQT_PLPjPA9gF0KmbOc=MXnRipqG_nAwt3sAGHQ@mail.gmail.com>
References: <A4577C7A-0798-4B23-A99E-55C5DFBF3F9F@segment7.net>
	<CA+TZ5hBfD_U0zBnz0pJF695teigMaFyvGJx-WtScHgNyjGmA-g@mail.gmail.com>
	<60F1B01D-C1F5-45F8-8C33-2C2F15BF91BE@segment7.net>
	<CA+TZ5hDNA79SHCO3Jvv6bqNdUh0LGyWWnKZuStBK6F8nr0wPUg@mail.gmail.com>
	<2549C44D-03C7-418F-B558-FE6F6BD4A38D@segment7.net>
	<CA+TZ5hA5=Tcx4Gk2T1WDMcjPsAEUuCKFMQbiy7=5rkxAD6676g@mail.gmail.com>
	<CACYs=59XQw0GuHR-xtGCR02PGSFvt36ztHycQT90uRb7GuZC1Q@mail.gmail.com>
	<CA+TZ5hDKS5NYg0JsBSwtTkmYrm7FwGRUOudk+myNw+AMK_6z0w@mail.gmail.com>
	<CA+TZ5hCUiVTzQT_PLPjPA9gF0KmbOc=MXnRipqG_nAwt3sAGHQ@mail.gmail.com>
Message-ID: <CACYs=58LvYJ9pVqLcsTtO4x7-+zSUO0wwrA35ueYpDcFz+LXEw@mail.gmail.com>

The cat's shredded the bag.  it's out.   I'm a scrupulous security
professional, but like he said, it's a public list. You can guarantee that
somebody on the list is using your access already and exploring what a
raps.org is.  If what it is needs to be secure then you need to be hurrying
with your password change.

You're not getting out of it.   If you've messed up then you admit your
mistake and work to correct it.   as a hiring/firing manager I can say: Do
that and everything goes well, try to cover it up and get your poop smelled
later and it's likely you're fired.


On Wed, May 16, 2012 at 4:54 PM, Kevin Kornemann <kevin at proofimaging.com>wrote:

> If possible, can you delete the email in your inbox? I mean surely a
> posted can be deleted somehow?
>
> On Wed, May 16, 2012 at 4:50 PM, Kevin Kornemann <kevin at proofimaging.com>wrote:
>
>> changed, but didn't get the rest.
>>
>>
>> On Wed, May 16, 2012 at 4:47 PM, Brian Kennedy <sirbeep at gmail.com> wrote:
>>
>>> It's in my email inbox,  so nope.  :)
>>>
>>> Cat's already shredded the bag and is working on the curtains.  Change
>>> your password quick, you don't know who's on the list that may have taken
>>> it and is running with it already.
>>>
>>>
>>> On Wed, May 16, 2012 at 4:38 PM, Kevin Kornemann <kevin at proofimaging.com
>>> > wrote:
>>>
>>>> shit, didn't realize this was public. Can I delete this post?
>>>>
>>>>
>>>> On Wed, May 16, 2012 at 4:25 PM, Eric Hodel <drbrain at segment7.net>wrote:
>>>>
>>>>> On May 16, 2012, at 11:35, Kevin Kornemann wrote:
>>>>> > cool, thanks! I'm trying to scrape a page that requires login, and
>>>>> then clicking on form check boxes and then hitting search. I managed to get
>>>>> this far with mechanize but the search results I get are garbage (not what
>>>>> I get when I do the same with a browser), so I was grasping at straws that
>>>>> it was a js script not being executed problem, but I really don't know.
>>>>> Looking into other options but realy nothing I found was as easy or nice to
>>>>> use as mechanize.
>>>>> >
>>>>> > If it's not too much trouble, any hints are appreciated... to
>>>>> re-woord my problem, search results are not as expected, even wrong page is
>>>>> being returned from what I can tell. This is my script:
>>>>> >
>>>>> > require 'nokogiri'
>>>>> > require 'mechanize'
>>>>> >
>>>>> > agent = Mechanize.new
>>>>> >
>>>>> > agent.follow_meta_refresh = true
>>>>> > agent.user_agent_alias = 'Mac Safari'
>>>>> >
>>>>> > page = agent.get('
>>>>> http://connect.raps.org/RAPS/Directory/AdvancedSearch/')
>>>>> >
>>>>> > login_form = page.form()
>>>>> >
>>>>> > login_form.LoginTextBox = 'alexa coppola'
>>>>> > login_form.PasswordTextBox = 'goodfriend123'
>>>>> >
>>>>> > page = agent.submit(login_form, login_form.buttons.first)
>>>>> >
>>>>> > agent.follow_meta_refresh = false
>>>>> >
>>>>> > search_form = page.form()
>>>>> >
>>>>> > search_form.checkbox_with(:name =>
>>>>> 'ctl00$MainCopy$ctl08$DemographicsList$0').check
>>>>> >
>>>>> > page = agent.submit(search_form, search_form.buttons.first)
>>>>> >
>>>>> >
>>>>> > # returns different page than expected, not search results.
>>>>>
>>>>> You're going to need to emulate what the javascript does when you
>>>>> click on the request page.  You can use browser debugging tools to compare
>>>>> what query a browser makes with what query mechanize makes (through use of
>>>>> the debugger).
>>>>>
>>>>> PS:  This is a public list so you will want to change the password.
>>>>> _______________________________________________
>>>>> Mechanize-users mailing list
>>>>> Mechanize-users at rubyforge.org
>>>>> http://rubyforge.org/mailman/listinfo/mechanize-users
>>>>>
>>>>
>>>>
>>>> _______________________________________________
>>>> Mechanize-users mailing list
>>>> Mechanize-users at rubyforge.org
>>>> http://rubyforge.org/mailman/listinfo/mechanize-users
>>>>
>>>
>>>
>>> _______________________________________________
>>> Mechanize-users mailing list
>>> Mechanize-users at rubyforge.org
>>> http://rubyforge.org/mailman/listinfo/mechanize-users
>>>
>>
>>
>
> _______________________________________________
> Mechanize-users mailing list
> Mechanize-users at rubyforge.org
> http://rubyforge.org/mailman/listinfo/mechanize-users
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://rubyforge.org/pipermail/mechanize-users/attachments/20120516/b409e1fa/attachment.html>

From godfreykfc at gmail.com  Wed May 16 20:58:36 2012
From: godfreykfc at gmail.com (Godfrey Chan)
Date: Wed, 16 May 2012 13:58:36 -0700
Subject: [Mechanize-users] [ANN] mechanize 2.5.1 Released
In-Reply-To: <CA+TZ5hCUiVTzQT_PLPjPA9gF0KmbOc=MXnRipqG_nAwt3sAGHQ@mail.gmail.com>
References: <A4577C7A-0798-4B23-A99E-55C5DFBF3F9F@segment7.net>
	<CA+TZ5hBfD_U0zBnz0pJF695teigMaFyvGJx-WtScHgNyjGmA-g@mail.gmail.com>
	<60F1B01D-C1F5-45F8-8C33-2C2F15BF91BE@segment7.net>
	<CA+TZ5hDNA79SHCO3Jvv6bqNdUh0LGyWWnKZuStBK6F8nr0wPUg@mail.gmail.com>
	<2549C44D-03C7-418F-B558-FE6F6BD4A38D@segment7.net>
	<CA+TZ5hA5=Tcx4Gk2T1WDMcjPsAEUuCKFMQbiy7=5rkxAD6676g@mail.gmail.com>
	<CACYs=59XQw0GuHR-xtGCR02PGSFvt36ztHycQT90uRb7GuZC1Q@mail.gmail.com>
	<CA+TZ5hDKS5NYg0JsBSwtTkmYrm7FwGRUOudk+myNw+AMK_6z0w@mail.gmail.com>
	<CA+TZ5hCUiVTzQT_PLPjPA9gF0KmbOc=MXnRipqG_nAwt3sAGHQ@mail.gmail.com>
Message-ID: <FF4F523B-8E63-4D3A-857A-6E69C9400337@gmail.com>

I don't think you understand how a mailing list works. Basically you just sent an *email* out to hundreds of email addresses. So the short answer is, no.

Godfrey

On 2012-05-16, at 1:54 PM, Kevin Kornemann wrote:

> If possible, can you delete the email in your inbox? I mean surely a posted can be deleted somehow?
> 
> On Wed, May 16, 2012 at 4:50 PM, Kevin Kornemann <kevin at proofimaging.com> wrote:
> changed, but didn't get the rest.
> 
> 
> On Wed, May 16, 2012 at 4:47 PM, Brian Kennedy <sirbeep at gmail.com> wrote:
> It's in my email inbox,  so nope.  :)
> 
> Cat's already shredded the bag and is working on the curtains.  Change your password quick, you don't know who's on the list that may have taken it and is running with it already.
> 
> 
> On Wed, May 16, 2012 at 4:38 PM, Kevin Kornemann <kevin at proofimaging.com> wrote:
> shit, didn't realize this was public. Can I delete this post?
> 
> 
> On Wed, May 16, 2012 at 4:25 PM, Eric Hodel <drbrain at segment7.net> wrote:
> On May 16, 2012, at 11:35, Kevin Kornemann wrote:
> > cool, thanks! I'm trying to scrape a page that requires login, and then clicking on form check boxes and then hitting search. I managed to get this far with mechanize but the search results I get are garbage (not what I get when I do the same with a browser), so I was grasping at straws that it was a js script not being executed problem, but I really don't know. Looking into other options but realy nothing I found was as easy or nice to use as mechanize.
> >
> > If it's not too much trouble, any hints are appreciated... to re-woord my problem, search results are not as expected, even wrong page is being returned from what I can tell. This is my script:
> >
> > require 'nokogiri'
> > require 'mechanize'
> >
> > agent = Mechanize.new
> >
> > agent.follow_meta_refresh = true
> > agent.user_agent_alias = 'Mac Safari'
> >
> > page = agent.get('http://connect.raps.org/RAPS/Directory/AdvancedSearch/')
> >
> > login_form = page.form()
> >
> > login_form.LoginTextBox = 'alexa coppola'
> > login_form.PasswordTextBox = 'goodfriend123'
> >
> > page = agent.submit(login_form, login_form.buttons.first)
> >
> > agent.follow_meta_refresh = false
> >
> > search_form = page.form()
> >
> > search_form.checkbox_with(:name => 'ctl00$MainCopy$ctl08$DemographicsList$0').check
> >
> > page = agent.submit(search_form, search_form.buttons.first)
> >
> >
> > # returns different page than expected, not search results.
> 
> You're going to need to emulate what the javascript does when you click on the request page.  You can use browser debugging tools to compare what query a browser makes with what query mechanize makes (through use of the debugger).
> 
> PS:  This is a public list so you will want to change the password.
> _______________________________________________
> Mechanize-users mailing list
> Mechanize-users at rubyforge.org
> http://rubyforge.org/mailman/listinfo/mechanize-users
> 
> 
> _______________________________________________
> Mechanize-users mailing list
> Mechanize-users at rubyforge.org
> http://rubyforge.org/mailman/listinfo/mechanize-users
> 
> 
> _______________________________________________
> Mechanize-users mailing list
> Mechanize-users at rubyforge.org
> http://rubyforge.org/mailman/listinfo/mechanize-users
> 
> 
> _______________________________________________
> Mechanize-users mailing list
> Mechanize-users at rubyforge.org
> http://rubyforge.org/mailman/listinfo/mechanize-users

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://rubyforge.org/pipermail/mechanize-users/attachments/20120516/072b352d/attachment.html>

From drbrain at segment7.net  Wed May 16 21:05:16 2012
From: drbrain at segment7.net (Eric Hodel)
Date: Wed, 16 May 2012 14:05:16 -0700
Subject: [Mechanize-users] [ANN] mechanize 2.5.1 Released
In-Reply-To: <CA+TZ5hCUiVTzQT_PLPjPA9gF0KmbOc=MXnRipqG_nAwt3sAGHQ@mail.gmail.com>
References: <A4577C7A-0798-4B23-A99E-55C5DFBF3F9F@segment7.net>
	<CA+TZ5hBfD_U0zBnz0pJF695teigMaFyvGJx-WtScHgNyjGmA-g@mail.gmail.com>
	<60F1B01D-C1F5-45F8-8C33-2C2F15BF91BE@segment7.net>
	<CA+TZ5hDNA79SHCO3Jvv6bqNdUh0LGyWWnKZuStBK6F8nr0wPUg@mail.gmail.com>
	<2549C44D-03C7-418F-B558-FE6F6BD4A38D@segment7.net>
	<CA+TZ5hA5=Tcx4Gk2T1WDMcjPsAEUuCKFMQbiy7=5rkxAD6676g@mail.gmail.com>
	<CACYs=59XQw0GuHR-xtGCR02PGSFvt36ztHycQT90uRb7GuZC1Q@mail.gmail.com>
	<CA+TZ5hDKS5NYg0JsBSwtTkmYrm7FwGRUOudk+myNw+AMK_6z0w@mail.gmail.com>
	<CA+TZ5hCUiVTzQT_PLPjPA9gF0KmbOc=MXnRipqG_nAwt3sAGHQ@mail.gmail.com>
Message-ID: <7B6DF1B2-76D7-480C-825F-B11DB3AF2A17@segment7.net>

On May 16, 2012, at 13:54, Kevin Kornemann wrote:
> If possible, can you delete the email in your inbox? I mean surely a posted can be deleted somehow?

It's possible that google has already picked up the email from the list archive page. I'm unsure if there are any mirrors of this list, but it's possible your password has been transferred far beyond this list by now.

Changing the password should be sufficient.

From kevin at proofimaging.com  Wed May 16 21:12:45 2012
From: kevin at proofimaging.com (Kevin Kornemann)
Date: Wed, 16 May 2012 17:12:45 -0400
Subject: [Mechanize-users] [ANN] mechanize 2.5.1 Released
In-Reply-To: <FF4F523B-8E63-4D3A-857A-6E69C9400337@gmail.com>
References: <A4577C7A-0798-4B23-A99E-55C5DFBF3F9F@segment7.net>
	<CA+TZ5hBfD_U0zBnz0pJF695teigMaFyvGJx-WtScHgNyjGmA-g@mail.gmail.com>
	<60F1B01D-C1F5-45F8-8C33-2C2F15BF91BE@segment7.net>
	<CA+TZ5hDNA79SHCO3Jvv6bqNdUh0LGyWWnKZuStBK6F8nr0wPUg@mail.gmail.com>
	<2549C44D-03C7-418F-B558-FE6F6BD4A38D@segment7.net>
	<CA+TZ5hA5=Tcx4Gk2T1WDMcjPsAEUuCKFMQbiy7=5rkxAD6676g@mail.gmail.com>
	<CACYs=59XQw0GuHR-xtGCR02PGSFvt36ztHycQT90uRb7GuZC1Q@mail.gmail.com>
	<CA+TZ5hDKS5NYg0JsBSwtTkmYrm7FwGRUOudk+myNw+AMK_6z0w@mail.gmail.com>
	<CA+TZ5hCUiVTzQT_PLPjPA9gF0KmbOc=MXnRipqG_nAwt3sAGHQ@mail.gmail.com>
	<FF4F523B-8E63-4D3A-857A-6E69C9400337@gmail.com>
Message-ID: <CA+TZ5hAqE6-hPJttkvU=6Zf-njhoH5mcoX8oXx_c8imBYRSZzg@mail.gmail.com>

right, sorry.

On Wed, May 16, 2012 at 4:58 PM, Godfrey Chan <godfreykfc at gmail.com> wrote:

> I don't think you understand how a mailing list works. Basically you just
> sent an *email* out to hundreds of email addresses. So the short answer is,
> no.
>
> Godfrey
>
> On 2012-05-16, at 1:54 PM, Kevin Kornemann wrote:
>
> If possible, can you delete the email in your inbox? I mean surely a
> posted can be deleted somehow?
>
> On Wed, May 16, 2012 at 4:50 PM, Kevin Kornemann <kevin at proofimaging.com>wrote:
>
>> changed, but didn't get the rest.
>>
>>
>> On Wed, May 16, 2012 at 4:47 PM, Brian Kennedy <sirbeep at gmail.com> wrote:
>>
>>> It's in my email inbox,  so nope.  :)
>>>
>>> Cat's already shredded the bag and is working on the curtains.  Change
>>> your password quick, you don't know who's on the list that may have taken
>>> it and is running with it already.
>>>
>>>
>>> On Wed, May 16, 2012 at 4:38 PM, Kevin Kornemann <kevin at proofimaging.com
>>> > wrote:
>>>
>>>> shit, didn't realize this was public. Can I delete this post?
>>>>
>>>>
>>>> On Wed, May 16, 2012 at 4:25 PM, Eric Hodel <drbrain at segment7.net>wrote:
>>>>
>>>>> On May 16, 2012, at 11:35, Kevin Kornemann wrote:
>>>>> > cool, thanks! I'm trying to scrape a page that requires login, and
>>>>> then clicking on form check boxes and then hitting search. I managed to get
>>>>> this far with mechanize but the search results I get are garbage (not what
>>>>> I get when I do the same with a browser), so I was grasping at straws that
>>>>> it was a js script not being executed problem, but I really don't know.
>>>>> Looking into other options but realy nothing I found was as easy or nice to
>>>>> use as mechanize.
>>>>> >
>>>>> > If it's not too much trouble, any hints are appreciated... to
>>>>> re-woord my problem, search results are not as expected, even wrong page is
>>>>> being returned from what I can tell. This is my script:
>>>>> >
>>>>> > require 'nokogiri'
>>>>> > require 'mechanize'
>>>>> >
>>>>> > agent = Mechanize.new
>>>>> >
>>>>> > agent.follow_meta_refresh = true
>>>>> > agent.user_agent_alias = 'Mac Safari'
>>>>> >
>>>>> > page = agent.get('
>>>>> http://connect.raps.org/RAPS/Directory/AdvancedSearch/')
>>>>> >
>>>>> > login_form = page.form()
>>>>> >
>>>>> > login_form.LoginTextBox = 'alexa coppola'
>>>>> > login_form.PasswordTextBox = 'goodfriend123'
>>>>> >
>>>>> > page = agent.submit(login_form, login_form.buttons.first)
>>>>> >
>>>>> > agent.follow_meta_refresh = false
>>>>> >
>>>>> > search_form = page.form()
>>>>> >
>>>>> > search_form.checkbox_with(:name =>
>>>>> 'ctl00$MainCopy$ctl08$DemographicsList$0').check
>>>>> >
>>>>> > page = agent.submit(search_form, search_form.buttons.first)
>>>>> >
>>>>> >
>>>>> > # returns different page than expected, not search results.
>>>>>
>>>>> You're going to need to emulate what the javascript does when you
>>>>> click on the request page.  You can use browser debugging tools to compare
>>>>> what query a browser makes with what query mechanize makes (through use of
>>>>> the debugger).
>>>>>
>>>>> PS:  This is a public list so you will want to change the password.
>>>>> _______________________________________________
>>>>> Mechanize-users mailing list
>>>>> Mechanize-users at rubyforge.org
>>>>> http://rubyforge.org/mailman/listinfo/mechanize-users
>>>>>
>>>>
>>>>
>>>> _______________________________________________
>>>> Mechanize-users mailing list
>>>> Mechanize-users at rubyforge.org
>>>> http://rubyforge.org/mailman/listinfo/mechanize-users
>>>>
>>>
>>>
>>> _______________________________________________
>>> Mechanize-users mailing list
>>> Mechanize-users at rubyforge.org
>>> http://rubyforge.org/mailman/listinfo/mechanize-users
>>>
>>
>>
> _______________________________________________
> Mechanize-users mailing list
> Mechanize-users at rubyforge.org
> http://rubyforge.org/mailman/listinfo/mechanize-users
>
>
>
> _______________________________________________
> Mechanize-users mailing list
> Mechanize-users at rubyforge.org
> http://rubyforge.org/mailman/listinfo/mechanize-users
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://rubyforge.org/pipermail/mechanize-users/attachments/20120516/75ae735f/attachment-0001.html>

From jack at groundbreakingsoftware.com  Wed May 23 19:40:15 2012
From: jack at groundbreakingsoftware.com (Jack Royal-Gordon)
Date: Wed, 23 May 2012 12:40:15 -0700
Subject: [Mechanize-users] Error trying to reload serialized page
Message-ID: <A58672F4-39E4-499F-9285-223CF22660F7@GroundbreakingSoftware.com>

What is the correct way to serialize an HTML page gotten from Agent::get, and then to reload the serialized page back into Mechanize?  I tried the following code:
	agent = Mechanize.new
	p1 = agent.get(url)
	=> #<Mechanize::Page ? >

	p2 = Mechanize::Page.new(p1.uri.to_s, p1.response.to_s, p1.body)

and got the error:
	Mechanize::ContentTypeError Exception: Mechanize::ContentTypeError

I also tried:
	p2 = Mechanize::Page .new(p1.uri, p1.response, p1.body)

and got the error:
	INTERNAL ERROR!!! undefined method `follow_meta_refresh' for nil:NilClass
	/Users/jackrg/Documents/Novelty-Stats/vendor/local/ruby/1.9.1/gems/activesupport-3.1.3/lib/active_support/whiny_nil.rb:48:in `method_missing'
	/Users/jackrg/Documents/Novelty-Stats/vendor/local/ruby/1.9.1/gems/mechanize-2.4/lib/mechanize/page.rb:350:in `meta_refresh'
	/Users/jackrg/Documents/Novelty-Stats/vendor/local/ruby/1.9.1/gems/mechanize-2.4/lib/mechanize/page.rb:139:in `block (2 levels) in pretty_print'
	/Users/jackrg/.rbenv/versions/1.9.2-p290/lib/ruby/1.9.1/prettyprint.rb:199:in `block (2 levels) in group'
	/Users/jackrg/.rbenv/versions/1.9.2-p290/lib/ruby/1.9.1/prettyprint.rb:225:in `nest'
	/Users/jackrg/.rbenv/versions/1.9.2-p290/lib/ruby/1.9.1/prettyprint.rb:198:in `block in group'
	/Users/jackrg/.rbenv/versions/1.9.2-p290/lib/ruby/1.9.1/prettyprint.rb:210:in `group_sub'
	/Users/jackrg/.rbenv/versions/1.9.2-p290/lib/ruby/1.9.1/prettyprint.rb:197:in `group'
	/Users/jackrg/Documents/Novelty-Stats/vendor/local/ruby/1.9.1/gems/mechanize-2.4/lib/mechanize/page.rb:138:in `block in pretty_print'
	/Users/jackrg/.rbenv/versions/1.9.2-p290/lib/ruby/1.9.1/prettyprint.rb:199:in `block (2 levels) in group'
	/Users/jackrg/.rbenv/versions/1.9.2-p290/lib/ruby/1.9.1/prettyprint.rb:225:in `nest'
	/Users/jackrg/.rbenv/versions/1.9.2-p290/lib/ruby/1.9.1/prettyprint.rb:198:in `block in group'
	/Users/jackrg/.rbenv/versions/1.9.2-p290/lib/ruby/1.9.1/prettyprint.rb:210:in `group_sub'
	/Users/jackrg/.rbenv/versions/1.9.2-p290/lib/ruby/1.9.1/prettyprint.rb:197:in `group'
	/Users/jackrg/.rbenv/versions/1.9.2-p290/lib/ruby/1.9.1/pp.rb:164:in `object_group'
	/Users/jackrg/Documents/Novelty-Stats/vendor/local/ruby/1.9.1/gems/mechanize-2.4/lib/mechanize/page.rb:134:in `pretty_print'
	/Users/jackrg/.rbenv/versions/1.9.2-p290/lib/ruby/1.9.1/pp.rb:154:in `block in pp'
	/Users/jackrg/.rbenv/versions/1.9.2-p290/lib/ruby/1.9.1/prettyprint.rb:199:in `block (2 levels) in group'
	/Users/jackrg/.rbenv/versions/1.9.2-p290/lib/ruby/1.9.1/prettyprint.rb:225:in `nest'
	/Users/jackrg/.rbenv/versions/1.9.2-p290/lib/ruby/1.9.1/prettyprint.rb:198:in `block in group'
	/Users/jackrg/.rbenv/versions/1.9.2-p290/lib/ruby/1.9.1/prettyprint.rb:210:in `group_sub'
	/Users/jackrg/.rbenv/versions/1.9.2-p290/lib/ruby/1.9.1/prettyprint.rb:197:in `group'
	/Users/jackrg/.rbenv/versions/1.9.2-p290/lib/ruby/1.9.1/pp.rb:154:in `pp'
	/Users/jackrg/.rbenv/versions/1.9.2-p290/lib/ruby/1.9.1/pp.rb:77:in `block in pp'
	/Users/jackrg/.rbenv/versions/1.9.2-p290/lib/ruby/1.9.1/pp.rb:121:in `guard_inspect_key'
	/Users/jackrg/.rbenv/versions/1.9.2-p290/lib/ruby/1.9.1/pp.rb:77:in `pp'
	/Users/jackrg/.rbenv/versions/1.9.2-p290/lib/ruby/1.9.1/pp.rb:51:in `pretty_inspect'
	/Users/jackrg/Documents/Novelty-Stats/vendor/local/ruby/1.9.1/gems/ruby-debug19-0.11.6/cli/ruby-debug/commands/eval.rb:47:in `block in execute'
	/Users/jackrg/Documents/Novelty-Stats/vendor/local/ruby/1.9.1/gems/ruby-debug19-0.11.6/cli/ruby-debug/commands/eval.rb:19:in `run_with_binding'
	/Users/jackrg/Documents/Novelty-Stats/vendor/local/ruby/1.9.1/gems/ruby-debug19-0.11.6/cli/ruby-debug/commands/eval.rb:46:in `execute'
	/Users/jackrg/Documents/Novelty-Stats/vendor/local/ruby/1.9.1/gems/ruby-debug19-0.11.6/cli/ruby-debug/processor.rb:280:in `one_cmd'
	/Users/jackrg/Documents/Novelty-Stats/vendor/local/ruby/1.9.1/gems/ruby-debug19-0.11.6/cli/ruby-debug/processor.rb:261:in `block (2 levels) in process_commands'
	/Users/jackrg/Documents/Novelty-Stats/vendor/local/ruby/1.9.1/gems/ruby-debug19-0.11.6/cli/ruby-debug/processor.rb:260:in `each'
	/Users/jackrg/Documents/Novelty-Stats/vendor/local/ruby/1.9.1/gems/ruby-debug19-0.11.6/cli/ruby-debug/processor.rb:260:in `block in process_commands'
	/Users/jackrg/Documents/Novelty-Stats/vendor/local/ruby/1.9.1/gems/ruby-debug19-0.11.6/cli/ruby-debug/processor.rb:253:in `catch'
	/Users/jackrg/Documents/Novelty-Stats/vendor/local/ruby/1.9.1/gems/ruby-debug19-0.11.6/cli/ruby-debug/processor.rb:253:in `process_commands'
	/Users/jackrg/Documents/Novelty-Stats/vendor/local/ruby/1.9.1/gems/ruby-debug19-0.11.6/cli/ruby-debug/processor.rb:173:in `at_line'
	(eval):5:in `block in at_line'
	<internal:prelude>:10:in `synchronize'
	(eval):3:in `at_line'
	/Users/jackrg/Documents/Novelty-Stats/vendor/local/ruby/1.9.1/gems/ruby-debug-base19-0.11.25/lib/ruby-debug-base.rb:55:in `at_line'
	/Users/jackrg/Documents/Novelty-Stats/prototype/test4.rb:61:in `mechanize_page'
	(irb):4:in `irb_binding'


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://rubyforge.org/pipermail/mechanize-users/attachments/20120523/461b6f29/attachment.html>

From drbrain at segment7.net  Wed May 23 20:24:16 2012
From: drbrain at segment7.net (Eric Hodel)
Date: Wed, 23 May 2012 13:24:16 -0700
Subject: [Mechanize-users] Error trying to reload serialized page
In-Reply-To: <A58672F4-39E4-499F-9285-223CF22660F7@GroundbreakingSoftware.com>
References: <A58672F4-39E4-499F-9285-223CF22660F7@GroundbreakingSoftware.com>
Message-ID: <6971CA8D-AD31-4019-806C-C6E7CF7E8060@segment7.net>

On May 23, 2012, at 12:40, Jack Royal-Gordon wrote:
> What is the correct way to serialize an HTML page gotten from Agent::get, and then to reload the serialized page back into Mechanize?  I tried the following code:
> 	agent = Mechanize.new
> 	p1 = agent.get(url)
> 	=> #<Mechanize::Page ? >
> 
> 	p2 = Mechanize::Page.new(p1.uri.to_s, p1.response.to_s, p1.body)
> 
> and got the error:
> 	Mechanize::ContentTypeError Exception: Mechanize::ContentTypeError

A String is not an acceptable substitute for a Net::HTTPResponse or Hash (representing the headers of a response).

> I also tried:
> 	p2 = Mechanize::Page .new(p1.uri, p1.response, p1.body)

Try:

  p2 = Mechanize::Page.new p1.uri, p1.response, p1.body, agent

While Mechanize::Page defaults all these to nil, this doesn't actually work for Page.  I'll see what I can do to fix this.

> and got the error:
> 	INTERNAL ERROR!!! undefined method `follow_meta_refresh' for nil:NilClass
> 	/Users/jackrg/Documents/Novelty-Stats/vendor/local/ruby/1.9.1/gems/activesupport-3.1.3/lib/active_support/whiny_nil.rb:48:in `method_missing'
> 	/Users/jackrg/Documents/Novelty-Stats/vendor/local/ruby/1.9.1/gems/mechanize-2.4/lib/mechanize/page.rb:350:in `meta_refresh'

A Mechanize instance is consulted to determine how <meta> tags should be found, so you must pass it in.