From jasoninclass at googlemail.com Wed Dec 9 17:58:14 2009 From: jasoninclass at googlemail.com (jason franklin-stokes) Date: Wed, 9 Dec 2009 23:58:14 +0100 Subject: [Celerity-users] memory leaks and serialize blockage Message-ID: the method enable_event_listener has 2 side effects. 1. it seems to hold on to objects which result in the javascript threads not being able to close. so over time memory useage grows and cannot be collected by the GC 2. as soon as enable_event_listener has been called it becomes no longer possible to serilaize webclient. question. what exactly is this method for? is there another way of doing this so that it does hold objects in memory and does not get in the way of serialize? would be great to get some feedback thanks jason. From tomasz2k at poczta.onet.pl Wed Dec 9 18:08:14 2009 From: tomasz2k at poczta.onet.pl (Tomasz =?utf-8?q?Kalkosi=C5=84ski?=) Date: Thu, 10 Dec 2009 00:08:14 +0100 Subject: [Celerity-users] memory leaks and serialize blockage In-Reply-To: References: Message-ID: <200912100008.14373.tomasz2k@poczta.onet.pl> On Wednesday 09 of December 2009 23:58:14 jason franklin-stokes wrote: > > the method enable_event_listener has 2 side effects. > > 1. it seems to hold on to objects which result in the javascript threads not being able to close. so over time memory useage grows and cannot be collected by the GC > > 2. as soon as enable_event_listener has been called it becomes no longer possible to serilaize webclient. > > question. > > what exactly is this method for? is there another way of doing this so that it does hold objects in memory and does not get in the way of serialize? > > would be great to get some feedback Hello jason I use Celerity with spawned 10 threads and I saw memory consuming is really high in here too. Can you provide some tests for your assumptions about enable_event_listener? I don't know the real purpose of this method. Greetings, Tomasz Kalkosi?ski > > thanks > > jason. > > _______________________________________________ > Celerity-users mailing list > Celerity-users at rubyforge.org > http://rubyforge.org/mailman/listinfo/celerity-users > From jasoninclass at googlemail.com Thu Dec 10 02:24:53 2009 From: jasoninclass at googlemail.com (jason franklin-stokes) Date: Thu, 10 Dec 2009 08:24:53 +0100 Subject: [Celerity-users] memory leaks and serialize blockage In-Reply-To: <200912100008.14373.tomasz2k@poczta.onet.pl> References: <200912100008.14373.tomasz2k@poczta.onet.pl> Message-ID: Hi Tom, I can't provide tests at the moment - my hypotheses is based on observation. 1. I cannot serialize webclient in htmlunit when i have gone through celerity - but i can do it if i dont use celerity. 2. when i return directly from enable_event_listener (i.e. i dont let code run) the number of loose javascript threads goes down from 60 to 13 (over 200 websites) and 3. the time needed to reach memory limit is about 2 times longer. one thing that looks suspicious to me is in the celerity listener.rb it looks like instances of weblient are being given objects that contain webclient. This is a loop reference for the garbage collector which it should be able to handle..... unless webclient is being used in a different thread - which I think it is in the case of javascript background tasks - which would mean that it cannot get garbage collected as long as the thread is running - and for some reason the thread wont exit on close all windows (my be this has something to do with the self referencing - I am just guessing here). Therefore we end up with a growing number of threads all holding their own webclient objects. The guys at htmluniit say they have almost entirely sure they have no memory leakage - and that all reports of memory leakage in the past months have been due to users not the htmlunit code. Now this is pretty big leakage and i am sure that they would have discovered a problem as big as this by now. So my guess is that the problem is most probably in the celerity code. here is the block of code that i think is causing the problem. def initialize(webclient) @webclient = webclient @procs = Hash.new { |h, k| h[k] = [] } end def add_listener(type, &block) case type when :status @webclient.setStatusHandler(self) when :alert @webclient.setAlertHandler(self) when :attachment @webclient.setAttachmentHandler(self) when :web_window_event @webclient.addWebWindowListener(self) when :html_parser @webclient.setHTMLParserListener(self) when :incorrectness @webclient.setIncorrectnessListener(self) when :confirm @webclient.setConfirmHandler(self) when :prompt @webclient.setPromptHandler(self) else raise ArgumentError, "unknown listener type #{type.inspect}" end @procs[type] << block end On Dec 10, 2009, at 12:08 AM, Tomasz Kalkosi?ski wrote: > On Wednesday 09 of December 2009 23:58:14 jason franklin-stokes wrote: >> >> the method enable_event_listener has 2 side effects. >> >> 1. it seems to hold on to objects which result in the javascript threads not being able to close. so over time memory useage grows and cannot be collected by the GC >> >> 2. as soon as enable_event_listener has been called it becomes no longer possible to serilaize webclient. >> >> question. >> >> what exactly is this method for? is there another way of doing this so that it does hold objects in memory and does not get in the way of serialize? >> >> would be great to get some feedback > > Hello jason > > I use Celerity with spawned 10 threads and I saw memory consuming is really high in here too. > > Can you provide some tests for your assumptions about enable_event_listener? I don't know the real purpose of this method. > > Greetings, > Tomasz Kalkosi?ski > >> >> thanks >> >> jason. >> >> _______________________________________________ >> Celerity-users mailing list >> Celerity-users at rubyforge.org >> http://rubyforge.org/mailman/listinfo/celerity-users >> > > _______________________________________________ > Celerity-users mailing list > Celerity-users at rubyforge.org > http://rubyforge.org/mailman/listinfo/celerity-users From jari.bakken at gmail.com Thu Dec 10 09:52:58 2009 From: jari.bakken at gmail.com (Jari Bakken) Date: Thu, 10 Dec 2009 15:52:58 +0100 Subject: [Celerity-users] memory leaks and serialize blockage In-Reply-To: References: <200912100008.14373.tomasz2k@poczta.onet.pl> Message-ID: On Wed, Dec 9, 2009 at 11:58 PM, jason franklin-stokes wrote: > > what exactly is this method for? is there another way of doing this so that it does hold objects in memory and does not get in the way of serialize? > Celerity listens for web window events so that it knows when the page has changed, and can update its internal @page reference accordingly. So it's pretty important. I'm not aware of another way to do this without breaking tests, e.g. pages like this: http://github.com/jarib/watirspec/blob/master/html/timeout_window_location.html If you can provide some code that shows this is the source of a memory leak, I'll be happy to work on it. Without some way of testing/observing, I'll just be wandering in the dark. If you think the Celerity::Listener class is at fault, you can try this and see if the situation improves: def enable_event_listener @webclient.addWebWindowListener do |event| self.page = @page ? @page.getEnclosingWindow.getEnclosedPage : event.getNewPage end end About the serialization issue: When we add a Java interface implementation written in JRuby as a listener to the WebClient, it's no longer serializabe. This is because JRuby's interface implementations are not serializable. I've explained this before: http://rubyforge.org/pipermail/celerity-users/2009-September/000175.html You should be able to serialize the WebClient/page after removing the event listener: listener = browser.send :listener c.webclient.removeWebWindowListener(listener) From jasoninclass at googlemail.com Thu Dec 10 11:05:00 2009 From: jasoninclass at googlemail.com (jason franklin-stokes) Date: Thu, 10 Dec 2009 17:05:00 +0100 Subject: [Celerity-users] memory leaks and serialize blockage In-Reply-To: References: <200912100008.14373.tomasz2k@poczta.onet.pl> Message-ID: <90462369-83D2-4898-8E3E-734CA5341BF1@googlemail.com> Hi Jari, I have localized the problem down to two methods: add_listener in the Lister class and click_and_attach in the Browser class I have removed / replaced the use of these two methods from my code and I have no more loose threads or memory leaks and i get the serialization of webclient to work properly. i obviously need the listeners and the click_and_attach method (i only use this method of clicking something). i will try an come up with a stable reproduction of the problem and let you know as soon i have that. best jason. On Dec 10, 2009, at 3:52 PM, Jari Bakken wrote: > On Wed, Dec 9, 2009 at 11:58 PM, jason franklin-stokes > wrote: >> >> what exactly is this method for? is there another way of doing this so that it does hold objects in memory and does not get in the way of serialize? >> > > Celerity listens for web window events so that it knows when the page > has changed, and can update its internal @page reference accordingly. > So it's pretty important. I'm not aware of another way to do this > without breaking tests, e.g. pages like this: > > http://github.com/jarib/watirspec/blob/master/html/timeout_window_location.html > > If you can provide some code that shows this is the source of a memory > leak, I'll be happy to work on it. Without some way of > testing/observing, I'll just be wandering in the dark. > > If you think the Celerity::Listener class is at fault, you can try > this and see if the situation improves: > > def enable_event_listener > @webclient.addWebWindowListener do |event| > self.page = @page ? @page.getEnclosingWindow.getEnclosedPage : > event.getNewPage > end > end > > About the serialization issue: > > When we add a Java interface implementation written in JRuby as a > listener to the WebClient, it's no longer serializabe. This is because > JRuby's interface implementations are not serializable. I've explained > this before: > > http://rubyforge.org/pipermail/celerity-users/2009-September/000175.html > > You should be able to serialize the WebClient/page after removing the > event listener: > > listener = browser.send :listener > c.webclient.removeWebWindowListener(listener) > _______________________________________________ > Celerity-users mailing list > Celerity-users at rubyforge.org > http://rubyforge.org/mailman/listinfo/celerity-users From jasoninclass at googlemail.com Thu Dec 10 14:19:17 2009 From: jasoninclass at googlemail.com (jason franklin-stokes) Date: Thu, 10 Dec 2009 20:19:17 +0100 Subject: [Celerity-users] memory leaks and serialize blockage In-Reply-To: References: <200912100008.14373.tomasz2k@poczta.onet.pl> Message-ID: <9BABE365-B900-403B-AC19-F9CC4118F2FA@googlemail.com> Jari, just thinking about without looking at the code again... click_and_attach is probably thowing away the old instance of the webclient while the javascipt thread is still running which means that it wont get garbage collected. closeallwindows needs to be called on the webclient instance to ensure that the javascript threads get terminated. I am just not sure what side effects that would have on the celerity code by calling close on the celerity browser instance before letting it go in click_and_attach. what do you think? jason. On Dec 10, 2009, at 3:52 PM, Jari Bakken wrote: > On Wed, Dec 9, 2009 at 11:58 PM, jason franklin-stokes > wrote: >> >> what exactly is this method for? is there another way of doing this so that it does hold objects in memory and does not get in the way of serialize? >> > > Celerity listens for web window events so that it knows when the page > has changed, and can update its internal @page reference accordingly. > So it's pretty important. I'm not aware of another way to do this > without breaking tests, e.g. pages like this: > > http://github.com/jarib/watirspec/blob/master/html/timeout_window_location.html > > If you can provide some code that shows this is the source of a memory > leak, I'll be happy to work on it. Without some way of > testing/observing, I'll just be wandering in the dark. > > If you think the Celerity::Listener class is at fault, you can try > this and see if the situation improves: > > def enable_event_listener > @webclient.addWebWindowListener do |event| > self.page = @page ? @page.getEnclosingWindow.getEnclosedPage : > event.getNewPage > end > end > > About the serialization issue: > > When we add a Java interface implementation written in JRuby as a > listener to the WebClient, it's no longer serializabe. This is because > JRuby's interface implementations are not serializable. I've explained > this before: > > http://rubyforge.org/pipermail/celerity-users/2009-September/000175.html > > You should be able to serialize the WebClient/page after removing the > event listener: > > listener = browser.send :listener > c.webclient.removeWebWindowListener(listener) > _______________________________________________ > Celerity-users mailing list > Celerity-users at rubyforge.org > http://rubyforge.org/mailman/listinfo/celerity-users From jari.bakken at gmail.com Thu Dec 10 14:39:02 2009 From: jari.bakken at gmail.com (Jari Bakken) Date: Thu, 10 Dec 2009 20:39:02 +0100 Subject: [Celerity-users] memory leaks and serialize blockage In-Reply-To: <9BABE365-B900-403B-AC19-F9CC4118F2FA@googlemail.com> References: <200912100008.14373.tomasz2k@poczta.onet.pl> <9BABE365-B900-403B-AC19-F9CC4118F2FA@googlemail.com> Message-ID: On Thu, Dec 10, 2009 at 8:19 PM, jason franklin-stokes wrote: > Jari, > > just thinking about without looking at the code again... > click_and_attach is probably thowing away the old instance of the webclient while the javascipt thread is still running which means that it wont get garbage collected. closeallwindows needs to be called on the webclient instance to ensure that the javascript threads get terminated. I am just not sure what side effects that would have on the celerity code by calling close on the celerity browser instance before letting it go in click_and_attach. > > what do you think? > Nope, click_and_attach is not throwing away any old instances. Calling Browser#close is the user's responsibility - Celerity can't know if the user still wants to use the browser. Remember, this is how click_and_attach is supposed to be used: b = Browser.new b.goto("http://...) popup_browser = b.button(:id, 'trigger-popup-window').click_and_attach popup_browser.text_field(:id, 'foo').set("foobar") # or whatever popup_browser.close b.button(:id, 'bar').click # or whatever - continues using the same browser If you could just show some code that exemplifies the problem you're having, I'm sure I'd be able to help. From jasoninclass at googlemail.com Thu Dec 10 15:28:22 2009 From: jasoninclass at googlemail.com (jason franklin-stokes) Date: Thu, 10 Dec 2009 21:28:22 +0100 Subject: [Celerity-users] memory leaks and serialize blockage In-Reply-To: References: <200912100008.14373.tomasz2k@poczta.onet.pl> <9BABE365-B900-403B-AC19-F9CC4118F2FA@googlemail.com> Message-ID: Jari, Thanks then it my fault, not celerites. I am not closing the browser after click_and_attach - so that solves that one. for some reason I thought that click_and_attach was sorting that out for me. I will put things back and see what happens. will let you know. BTW Here is the code for serialize and derserialize. This breaks when the listener is enabled. I added the following to the Celerity::Browser class include_class 'org.apache.commons.lang.SerializationUtils' def serialize_webclient SerializationUtils.serialize(@webclient).to_a.pack("C*") end def deserialize_webclient(_serialized_webclient) @webclient = SerializationUtils.deserialize(_serialized_webclient.to_java_bytes) end in the cli you can the do the following. browser = Celerity::Browser.new browser.goto "www.xyz.com" browser.serialize_webclient this will result in a java io not serializable exception if you make the Listner class method look like this: def add_listener(type, &block) return end and then run browser = Celerity::Browser.new browser.goto "www.xyz.com" browser.serialize_webclient it will work. I will recheck to see if the add_listner is also responsible for memory leaks and get back to you. jason. On Dec 10, 2009, at 8:39 PM, Jari Bakken wrote: > On Thu, Dec 10, 2009 at 8:19 PM, jason franklin-stokes > wrote: >> Jari, >> >> just thinking about without looking at the code again... >> click_and_attach is probably thowing away the old instance of the webclient while the javascipt thread is still running which means that it wont get garbage collected. closeallwindows needs to be called on the webclient instance to ensure that the javascript threads get terminated. I am just not sure what side effects that would have on the celerity code by calling close on the celerity browser instance before letting it go in click_and_attach. >> >> what do you think? >> > > Nope, click_and_attach is not throwing away any old instances. Calling > Browser#close is the user's responsibility - Celerity can't know if > the user still wants to use the browser. Remember, this is how > click_and_attach is supposed to be used: > > b = Browser.new > b.goto("http://...) > > popup_browser = b.button(:id, 'trigger-popup-window').click_and_attach > popup_browser.text_field(:id, 'foo').set("foobar") # or whatever > popup_browser.close > > b.button(:id, 'bar').click # or whatever - continues using the same browser > > If you could just show some code that exemplifies the problem you're > having, I'm sure I'd be able to help. > _______________________________________________ > Celerity-users mailing list > Celerity-users at rubyforge.org > http://rubyforge.org/mailman/listinfo/celerity-users From jari.bakken at gmail.com Thu Dec 10 15:44:46 2009 From: jari.bakken at gmail.com (Jari Bakken) Date: Thu, 10 Dec 2009 21:44:46 +0100 Subject: [Celerity-users] memory leaks and serialize blockage In-Reply-To: References: <200912100008.14373.tomasz2k@poczta.onet.pl> <9BABE365-B900-403B-AC19-F9CC4118F2FA@googlemail.com> Message-ID: On Thu, Dec 10, 2009 at 9:28 PM, jason franklin-stokes wrote: > > def add_listener(type, &block) > ? ? ?return > end > That is fine if you only use Browser#goto to navigate (assuming the page you goto doesn't do anything of importance in its onload handler). If you click links or interact with the page in other ways you're going to experience a lot of problems. Try running the watirspec suite to see what fails with that change. From jasoninclass at googlemail.com Thu Dec 10 15:57:50 2009 From: jasoninclass at googlemail.com (jason franklin-stokes) Date: Thu, 10 Dec 2009 21:57:50 +0100 Subject: [Celerity-users] memory leaks and serialize blockage In-Reply-To: References: <200912100008.14373.tomasz2k@poczta.onet.pl> <9BABE365-B900-403B-AC19-F9CC4118F2FA@googlemail.com> Message-ID: yep, i only did this for testing - i definitely need the listeners active - as much as i need serialization, so i guess i iam gonna have to find out what is happeing. there. On Dec 10, 2009, at 9:44 PM, Jari Bakken wrote: > On Thu, Dec 10, 2009 at 9:28 PM, jason franklin-stokes > wrote: >> >> def add_listener(type, &block) >> return >> end >> > > That is fine if you only use Browser#goto to navigate (assuming the > page you goto doesn't do anything of importance in its onload > handler). If you click links or interact with the page in other ways > you're going to experience a lot of problems. Try running the > watirspec suite to see what fails with that change. > _______________________________________________ > Celerity-users mailing list > Celerity-users at rubyforge.org > http://rubyforge.org/mailman/listinfo/celerity-users From jasoninclass at googlemail.com Thu Dec 10 17:28:57 2009 From: jasoninclass at googlemail.com (jason franklin-stokes) Date: Thu, 10 Dec 2009 23:28:57 +0100 Subject: [Celerity-users] memory leaks and serialize blockage In-Reply-To: References: <200912100008.14373.tomasz2k@poczta.onet.pl> <9BABE365-B900-403B-AC19-F9CC4118F2FA@googlemail.com> Message-ID: Jari, so serialize works when i take @web_client out of the initialize in Listener def initialize @procs = Hash.new { |h, k| h[k] = [] } end and add listener looks then like this: def add_listener(type, webclient, &block) case type when :status webclient.setStatusHandler(self) when :alert webclient.setAlertHandler(self) when :attachment webclient.setAttachmentHandler(self) when :web_window_event webclient.addWebWindowListener(self) when :html_parser webclient.setHTMLParserListener(self) when :incorrectness webclient.setIncorrectnessListener(self) when :confirm webclient.setConfirmHandler(self) when :prompt webclient.setPromptHandler(self) else raise ArgumentError, "unknown listener type #{type.inspect}" end @procs[type] << block end Obviously the callers have to pass webclient. I changed the required tests and they all pass - not perfect, but it works. BR Jason. On Dec 10, 2009, at 9:44 PM, Jari Bakken wrote: > On Thu, Dec 10, 2009 at 9:28 PM, jason franklin-stokes > wrote: >> >> def add_listener(type, &block) >> return >> end >> > > That is fine if you only use Browser#goto to navigate (assuming the > page you goto doesn't do anything of importance in its onload > handler). If you click links or interact with the page in other ways > you're going to experience a lot of problems. Try running the > watirspec suite to see what fails with that change. > _______________________________________________ > Celerity-users mailing list > Celerity-users at rubyforge.org > http://rubyforge.org/mailman/listinfo/celerity-users From jari.bakken at gmail.com Thu Dec 10 19:55:46 2009 From: jari.bakken at gmail.com (Jari Bakken) Date: Fri, 11 Dec 2009 01:55:46 +0100 Subject: [Celerity-users] memory leaks and serialize blockage In-Reply-To: References: <200912100008.14373.tomasz2k@poczta.onet.pl> <9BABE365-B900-403B-AC19-F9CC4118F2FA@googlemail.com> Message-ID: On Thu, Dec 10, 2009 at 11:28 PM, jason franklin-stokes wrote: > Jari, > > so serialize works when i take @web_client out of the initialize in Listener I'd be very surprised if that was the case. Are you sure you didn't change anything else? I tried making the same change and I still get: org/apache/commons/lang/SerializationUtils.java:111:in `serialize': org.apache.commons.lang.SerializationException: java.io.NotSerializableException: org.jruby.gen.InterfaceImpl874503039 (NativeException) from org/apache/commons/lang/SerializationUtils.java:133:in `serialize' from test.rb:7 Since the JRuby interface implementation is still being added to the WebClient, that makes total sense. From dorikick at gmail.com Thu Dec 10 22:34:26 2009 From: dorikick at gmail.com (doridori Jo) Date: Thu, 10 Dec 2009 19:34:26 -0800 Subject: [Celerity-users] memory leaks and serialize blockage In-Reply-To: References: <200912100008.14373.tomasz2k@poczta.onet.pl> <9BABE365-B900-403B-AC19-F9CC4118F2FA@googlemail.com> Message-ID: <3a8841390912101934v68ef97bq35a878fd8c715c71@mail.gmail.com> are you using the latest celerity ? 1.4.0 ? On 12/10/09, Jari Bakken wrote: > On Thu, Dec 10, 2009 at 11:28 PM, jason franklin-stokes > wrote: >> Jari, >> >> so serialize works when i take @web_client out of the initialize in >> Listener > > I'd be very surprised if that was the case. Are you sure you didn't > change anything else? > I tried making the same change and I still get: > > org/apache/commons/lang/SerializationUtils.java:111:in `serialize': > org.apache.commons.lang.SerializationException: > java.io.NotSerializableException: org.jruby.gen.InterfaceImpl874503039 > (NativeException) > from org/apache/commons/lang/SerializationUtils.java:133:in `serialize' > from test.rb:7 > > Since the JRuby interface implementation is still being added to the > WebClient, that makes total sense. > _______________________________________________ > Celerity-users mailing list > Celerity-users at rubyforge.org > http://rubyforge.org/mailman/listinfo/celerity-users > From dorikick at gmail.com Thu Dec 10 22:35:35 2009 From: dorikick at gmail.com (doridori Jo) Date: Thu, 10 Dec 2009 19:35:35 -0800 Subject: [Celerity-users] memory leaks and serialize blockage In-Reply-To: <3a8841390912101934v68ef97bq35a878fd8c715c71@mail.gmail.com> References: <9BABE365-B900-403B-AC19-F9CC4118F2FA@googlemail.com> <3a8841390912101934v68ef97bq35a878fd8c715c71@mail.gmail.com> Message-ID: <3a8841390912101935h4933a913h2364d9e9514bb810@mail.gmail.com> oops i meant jruby. latest jruby. On 12/10/09, doridori Jo wrote: > are you using the latest celerity ? 1.4.0 ? > > On 12/10/09, Jari Bakken wrote: >> On Thu, Dec 10, 2009 at 11:28 PM, jason franklin-stokes >> wrote: >>> Jari, >>> >>> so serialize works when i take @web_client out of the initialize in >>> Listener >> >> I'd be very surprised if that was the case. Are you sure you didn't >> change anything else? >> I tried making the same change and I still get: >> >> org/apache/commons/lang/SerializationUtils.java:111:in `serialize': >> org.apache.commons.lang.SerializationException: >> java.io.NotSerializableException: org.jruby.gen.InterfaceImpl874503039 >> (NativeException) >> from org/apache/commons/lang/SerializationUtils.java:133:in `serialize' >> from test.rb:7 >> >> Since the JRuby interface implementation is still being added to the >> WebClient, that makes total sense. >> _______________________________________________ >> Celerity-users mailing list >> Celerity-users at rubyforge.org >> http://rubyforge.org/mailman/listinfo/celerity-users >> > From jasoninclass at googlemail.com Fri Dec 11 02:35:30 2009 From: jasoninclass at googlemail.com (jason franklin-stokes) Date: Fri, 11 Dec 2009 08:35:30 +0100 Subject: [Celerity-users] memory leaks and serialize blockage In-Reply-To: <3a8841390912101934v68ef97bq35a878fd8c715c71@mail.gmail.com> References: <200912100008.14373.tomasz2k@poczta.onet.pl> <9BABE365-B900-403B-AC19-F9CC4118F2FA@googlemail.com> <3a8841390912101934v68ef97bq35a878fd8c715c71@mail.gmail.com> Message-ID: yep jruby 1.4.0 On Dec 11, 2009, at 4:34 AM, doridori Jo wrote: > are you using the latest celerity ? 1.4.0 ? > > On 12/10/09, Jari Bakken wrote: >> On Thu, Dec 10, 2009 at 11:28 PM, jason franklin-stokes >> wrote: >>> Jari, >>> >>> so serialize works when i take @web_client out of the initialize in >>> Listener >> >> I'd be very surprised if that was the case. Are you sure you didn't >> change anything else? >> I tried making the same change and I still get: >> >> org/apache/commons/lang/SerializationUtils.java:111:in `serialize': >> org.apache.commons.lang.SerializationException: >> java.io.NotSerializableException: org.jruby.gen.InterfaceImpl874503039 >> (NativeException) >> from org/apache/commons/lang/SerializationUtils.java:133:in `serialize' >> from test.rb:7 >> >> Since the JRuby interface implementation is still being added to the >> WebClient, that makes total sense. >> _______________________________________________ >> Celerity-users mailing list >> Celerity-users at rubyforge.org >> http://rubyforge.org/mailman/listinfo/celerity-users >> > _______________________________________________ > Celerity-users mailing list > Celerity-users at rubyforge.org > http://rubyforge.org/mailman/listinfo/celerity-users From jasoninclass at googlemail.com Fri Dec 11 03:23:26 2009 From: jasoninclass at googlemail.com (jason franklin-stokes) Date: Fri, 11 Dec 2009 09:23:26 +0100 Subject: [Celerity-users] memory leaks and serialize blockage In-Reply-To: <3a8841390912101934v68ef97bq35a878fd8c715c71@mail.gmail.com> References: <200912100008.14373.tomasz2k@poczta.onet.pl> <9BABE365-B900-403B-AC19-F9CC4118F2FA@googlemail.com> <3a8841390912101934v68ef97bq35a878fd8c715c71@mail.gmail.com> Message-ID: <8FC66F24-DBF5-4F85-BDA9-D51691F2A558@googlemail.com> Jari, You are totally right - this makes no difference - I obviously had a fantom version on celerity laying around in my vendor directory that I was testing changes on - sorry sorry. so i re-did everything fresh and back to sqaure 1. a. serialize and deserialize works perfectly for webclient and page when listeners have not been set. b. since I added the browser close after click_and_attach there are no memory leaks - perfect. the question of the day i suppose is, what on earth is causing serialize to bombout when the listeners have been set. jason. On Dec 11, 2009, at 4:34 AM, doridori Jo wrote: > are you using the latest celerity ? 1.4.0 ? > > On 12/10/09, Jari Bakken wrote: >> On Thu, Dec 10, 2009 at 11:28 PM, jason franklin-stokes >> wrote: >>> Jari, >>> >>> so serialize works when i take @web_client out of the initialize in >>> Listener >> >> I'd be very surprised if that was the case. Are you sure you didn't >> change anything else? >> I tried making the same change and I still get: >> >> org/apache/commons/lang/SerializationUtils.java:111:in `serialize': >> org.apache.commons.lang.SerializationException: >> java.io.NotSerializableException: org.jruby.gen.InterfaceImpl874503039 >> (NativeException) >> from org/apache/commons/lang/SerializationUtils.java:133:in `serialize' >> from test.rb:7 >> >> Since the JRuby interface implementation is still being added to the >> WebClient, that makes total sense. >> _______________________________________________ >> Celerity-users mailing list >> Celerity-users at rubyforge.org >> http://rubyforge.org/mailman/listinfo/celerity-users >> > _______________________________________________ > Celerity-users mailing list > Celerity-users at rubyforge.org > http://rubyforge.org/mailman/listinfo/celerity-users From jasoninclass at googlemail.com Fri Dec 11 03:30:41 2009 From: jasoninclass at googlemail.com (jason franklin-stokes) Date: Fri, 11 Dec 2009 09:30:41 +0100 Subject: [Celerity-users] memory leaks and serialize blockage In-Reply-To: References: <200912100008.14373.tomasz2k@poczta.onet.pl> <9BABE365-B900-403B-AC19-F9CC4118F2FA@googlemail.com> Message-ID: <01709500-5408-48ED-92B7-BA24C0D500E4@googlemail.com> aha, Jari, I think i see the problem with serializable, the htmlunit listeners implement serialiazeable so could it be that when a jruby object block has been set to them that they cannot serialize because the block does not impliment serializable?. On Dec 11, 2009, at 1:55 AM, Jari Bakken wrote: > On Thu, Dec 10, 2009 at 11:28 PM, jason franklin-stokes > wrote: >> Jari, >> >> so serialize works when i take @web_client out of the initialize in Listener > > I'd be very surprised if that was the case. Are you sure you didn't > change anything else? > I tried making the same change and I still get: > > org/apache/commons/lang/SerializationUtils.java:111:in `serialize': > org.apache.commons.lang.SerializationException: > java.io.NotSerializableException: org.jruby.gen.InterfaceImpl874503039 > (NativeException) > from org/apache/commons/lang/SerializationUtils.java:133:in `serialize' > from test.rb:7 > > Since the JRuby interface implementation is still being added to the > WebClient, that makes total sense. > _______________________________________________ > Celerity-users mailing list > Celerity-users at rubyforge.org > http://rubyforge.org/mailman/listinfo/celerity-users From dorikick at gmail.com Fri Dec 11 03:40:47 2009 From: dorikick at gmail.com (doridori Jo) Date: Fri, 11 Dec 2009 00:40:47 -0800 Subject: [Celerity-users] celerity has trouble with nailgun Message-ID: <3a8841390912110040x26e387f4x69f3960f0c542302@mail.gmail.com> im not sure if this is widely known but http://kenai.com/projects/jruby/pages/JRubyWithNailgun NailGun speeds up jruby startup a great deal...i think it runs a single JVM, and listens for jruby scripts running... anyways, when i try run a celerity script, it seems to crash. the script just requires celerity, and "hello world" message. without celerity, nailgun works in speeding up jruby a lot...run time command to see. -------------- next part -------------- An HTML attachment was scrubbed... URL: From jari.bakken at gmail.com Fri Dec 11 08:06:10 2009 From: jari.bakken at gmail.com (Jari Bakken) Date: Fri, 11 Dec 2009 14:06:10 +0100 Subject: [Celerity-users] memory leaks and serialize blockage In-Reply-To: <8FC66F24-DBF5-4F85-BDA9-D51691F2A558@googlemail.com> References: <200912100008.14373.tomasz2k@poczta.onet.pl> <9BABE365-B900-403B-AC19-F9CC4118F2FA@googlemail.com> <3a8841390912101934v68ef97bq35a878fd8c715c71@mail.gmail.com> <8FC66F24-DBF5-4F85-BDA9-D51691F2A558@googlemail.com> Message-ID: <22D72847-C117-46AB-B089-85C230708103@gmail.com> Den 11. des. 2009 kl. 09.23 skrev jason franklin-stokes : > > > the question of the day i suppose is, what on earth is causing > serialize to bombout when the listeners have been set. As I've mentioned several times, JRuby interface implementations are not serializable. That's why. You'll have to remove the listener before you serialize. > > > jason. > > > > On Dec 11, 2009, at 4:34 AM, doridori Jo wrote: > >> are you using the latest celerity ? 1.4.0 ? >> >> On 12/10/09, Jari Bakken wrote: >>> On Thu, Dec 10, 2009 at 11:28 PM, jason franklin-stokes >>> wrote: >>>> Jari, >>>> >>>> so serialize works when i take @web_client out of the initialize in >>>> Listener >>> >>> I'd be very surprised if that was the case. Are you sure you didn't >>> change anything else? >>> I tried making the same change and I still get: >>> >>> org/apache/commons/lang/SerializationUtils.java:111:in `serialize': >>> org.apache.commons.lang.SerializationException: >>> java.io.NotSerializableException: >>> org.jruby.gen.InterfaceImpl874503039 >>> (NativeException) >>> from org/apache/commons/lang/SerializationUtils.java:133:in >>> `serialize' >>> from test.rb:7 >>> >>> Since the JRuby interface implementation is still being added to the >>> WebClient, that makes total sense. >>> _______________________________________________ >>> Celerity-users mailing list >>> Celerity-users at rubyforge.org >>> http://rubyforge.org/mailman/listinfo/celerity-users >>> >> _______________________________________________ >> Celerity-users mailing list >> Celerity-users at rubyforge.org >> http://rubyforge.org/mailman/listinfo/celerity-users > > _______________________________________________ > Celerity-users mailing list > Celerity-users at rubyforge.org > http://rubyforge.org/mailman/listinfo/celerity-users From jari.bakken at gmail.com Fri Dec 11 08:07:40 2009 From: jari.bakken at gmail.com (Jari Bakken) Date: Fri, 11 Dec 2009 14:07:40 +0100 Subject: [Celerity-users] celerity has trouble with nailgun In-Reply-To: <3a8841390912110040x26e387f4x69f3960f0c542302@mail.gmail.com> References: <3a8841390912110040x26e387f4x69f3960f0c542302@mail.gmail.com> Message-ID: <2079DAC0-CC4F-41C9-AB01-8215E9901DD0@gmail.com> Please post the complete error you get, including the full backtrace. Den 11. des. 2009 kl. 09.40 skrev doridori Jo : > im not sure if this is widely known but http://kenai.com/projects/jruby/pages/JRubyWithNailgun > > NailGun speeds up jruby startup a great deal...i think it runs a > single JVM, and listens for jruby scripts running... > > anyways, when i try run a celerity script, it seems to crash. the > script just requires celerity, and "hello world" message. > > without celerity, nailgun works in speeding up jruby a lot...run > time command to see. > _______________________________________________ > Celerity-users mailing list > Celerity-users at rubyforge.org > http://rubyforge.org/mailman/listinfo/celerity-users -------------- next part -------------- An HTML attachment was scrubbed... URL: From jasoninclass at googlemail.com Sat Dec 12 11:17:44 2009 From: jasoninclass at googlemail.com (jason franklin-stokes) Date: Sat, 12 Dec 2009 17:17:44 +0100 Subject: [Celerity-users] click_and_attach Celerity::Exception::UnknownObjectException: no page loaded Message-ID: <9F4BB19E-E340-45BB-81A1-6C39486E5A38@googlemail.com> DearJari, I am stuck again with click_and_attach. After each click and and attach I call close all windows to make sure that I have not loose javascript threads hanging around - unfortunately this seems to have a side effect. After the second click I get a Celerity::Exception::UnknownObjectException: no page loaded exception. I cant figure out why. this is the code - you can just copy and paste it to the cli as it to see the error. have you got any idea what is going wrong . Thanks a million for any ideas. Jason. browser=Celerity::Browser.new browser.goto "http://www.swisspost.com" browser.wait element = browser.elements_by_xpath("/html/body/div[2]/div/div[4]/div[1]/ul[2]/li[2]/a").first new_browser = element.click_and_attach browser.wait browser.webclient.closeAllWindows browser.url browser = new_browser element = browser.elements_by_xpath("/html/body/div[2]/div/div[4]/div[1]/ul[1]/li[1]/a").first new_browser = element.click_and_attach browser.wait browser.webclient.closeAllWindows browser = new_browser browser.url # Celerity::Exception::UnknownObjectException: no page loaded From jari.bakken at gmail.com Sat Dec 12 11:43:20 2009 From: jari.bakken at gmail.com (Jari Bakken) Date: Sat, 12 Dec 2009 17:43:20 +0100 Subject: [Celerity-users] click_and_attach Celerity::Exception::UnknownObjectException: no page loaded In-Reply-To: <9F4BB19E-E340-45BB-81A1-6C39486E5A38@googlemail.com> References: <9F4BB19E-E340-45BB-81A1-6C39486E5A38@googlemail.com> Message-ID: On Sat, Dec 12, 2009 at 5:17 PM, jason franklin-stokes wrote: > browser = new_browser > > browser.url ? ? # Celerity::Exception::UnknownObjectException: no page loaded > > You need to navigate somewhere (i.e. Browser#goto) before you call Browser#url. From jasoninclass at googlemail.com Sat Dec 12 13:54:49 2009 From: jasoninclass at googlemail.com (jason franklin-stokes) Date: Sat, 12 Dec 2009 19:54:49 +0100 Subject: [Celerity-users] click_and_attach Celerity::Exception::UnknownObjectException: no page loaded In-Reply-To: References: <9F4BB19E-E340-45BB-81A1-6C39486E5A38@googlemail.com> Message-ID: Hi Jari, Thanks for the quick feedback, but i don't quite understand..... as far as I can see the browser is somewhere - it just came back from a click_and_attach. This works code fine..... ================== browser_0=Celerity::Browser.new browser_0.goto "http://www.swisspost.com" browser_0.wait element = browser_0.elements_by_xpath("/html/body/div[2]/div/div[4]/div[1]/ul[2]/li[2]/a").first browser_1 = element.click_and_attach browser_0.wait browser_0.webclient.closeAllWindows puts browser_1.url element = browser_1.elements_by_xpath("/html/body/div[2]/div/div[4]/div[1]/ul[1]/li[1]/a").first browser_2 = element.click_and_attach browser_1.wait browser_1.webclient.closeAllWindows puts browser_2.url This code does not ============== NB. although the only difference is that i close all windows on the browser i was using before i continue using the new browser i got from click_and_attach browser_0=Celerity::Browser.new browser_0.goto "http://www.swisspost.com" browser_0.wait element = browser_0.elements_by_xpath("/html/body/div[2]/div/div[4]/div[1]/ul[2]/li[2]/a").first browser_1 = element.click_and_attach browser_0.wait #browser_0.webclient.closeAllWindows puts browser_1.url element = browser_1.elements_by_xpath("/html/body/div[2]/div/div[4]/div[1]/ul[1]/li[1]/a").first browser_2 = element.click_and_attach browser_1.wait #browser_1.webclient.closeAllWindows puts browser_2.url ------- thanks for any ideas..... jason. On Dec 12, 2009, at 5:43 PM, Jari Bakken wrote: > On Sat, Dec 12, 2009 at 5:17 PM, jason franklin-stokes > wrote: >> browser = new_browser >> >> browser.url # Celerity::Exception::UnknownObjectException: no page loaded >> >> > > You need to navigate somewhere (i.e. Browser#goto) before you call Browser#url. > _______________________________________________ > Celerity-users mailing list > Celerity-users at rubyforge.org > http://rubyforge.org/mailman/listinfo/celerity-users From jasoninclass at googlemail.com Sat Dec 12 18:59:32 2009 From: jasoninclass at googlemail.com (jason franklin-stokes) Date: Sun, 13 Dec 2009 00:59:32 +0100 Subject: [Celerity-users] click_and_attach Celerity::Exception::UnknownObjectException: no page loaded In-Reply-To: References: <9F4BB19E-E340-45BB-81A1-6C39486E5A38@googlemail.com> Message-ID: <1E8912D9-CE75-4728-A185-015990F1D467@googlemail.com> Dear Jari, fyi, i can get no further with click_and_attach - the way it behaves makes no sense to me. I have added a new method to browser which does the same thing as click and attach but without generating a new instance of browser - which feels wrong anyway: def click_and_open_in_new_window assert_exists_and_enabled rescue_status_code_exception { browser.page = @object.openLinkInNewWindow } end also the celerity browser url method is using a depreciated htmlunit method so i changed that too: def url assert_exists @page.getWebResponse.getRequestSettings.getUrl.toString end this now returns the correct url rendered by element clicked returned from the server. This way i don't have to acquire a new instance of browser and close all windows of the browser instance after calling click_and_attach - which does not seem to work correctly anyway - at least in my case where i click several links in sequence using click_and_attach. The following code now works and no htmlunit javascript threads are left hanging around - code is shorter and less stuff happening that can go wrong (new browser instances, cookie handlers being copied, and listeners being lost , i.e. having to be setup again etc etc. browser=Celerity::Browser.new browser.goto "http://www.swisspost.com" browser.wait element = browser.elements_by_xpath("/html/body/div[2]/div/div[4]/div[1]/ul[2]/li[2]/a").first element.click_and_open_in_new_window # instead of click_and_attach browser.wait puts browser.url # correct url is returned and and page is loaded element = browser_0.elements_by_xpath("/html/body/div[2]/div/div[4]/div[1]/ul[1]/li[1]/a").first element.click_and_open_in_new_window # instead of click_and_attach browser.wait puts browser.url # correct url is returned and page is loaded - would normally break here if click_and_attach was being used browser.close # only needs to be done here and not for use of each click_and_attach Jason. On Dec 12, 2009, at 5:43 PM, Jari Bakken wrote: > On Sat, Dec 12, 2009 at 5:17 PM, jason franklin-stokes > wrote: >> browser = new_browser >> >> browser.url # Celerity::Exception::UnknownObjectException: no page loaded >> >> > > You need to navigate somewhere (i.e. Browser#goto) before you call Browser#url. > _______________________________________________ > Celerity-users mailing list > Celerity-users at rubyforge.org > http://rubyforge.org/mailman/listinfo/celerity-users From markus.kohler at gmail.com Mon Dec 14 18:00:18 2009 From: markus.kohler at gmail.com (Markus Kohler) Date: Tue, 15 Dec 2009 00:00:18 +0100 Subject: [Celerity-users] Parallel running test? Message-ID: <771905290912141500v12088bcap2aa580641516916c@mail.gmail.com> Hi all, Does someone have simple example of a multi threaded test? Regards, Markus -------------- next part -------------- An HTML attachment was scrubbed... URL: From jasoninclass at googlemail.com Mon Dec 14 18:43:59 2009 From: jasoninclass at googlemail.com (jason franklin-stokes) Date: Tue, 15 Dec 2009 00:43:59 +0100 Subject: [Celerity-users] Parallel running test? In-Reply-To: <771905290912141500v12088bcap2aa580641516916c@mail.gmail.com> References: <771905290912141500v12088bcap2aa580641516916c@mail.gmail.com> Message-ID: <54036DAA-C49A-4E07-9FDC-5F42F4D95EF7@googlemail.com> hmm, I have 100 running in Parallel.... I can't say that I can show you how that works in a few lines of code. I needed to make some changes to htmlunit and to celerity to getting running properly (and I am still not too sure that is). I ended up using rails in multithreded mode to encapsulate the browser instances and a seperate server to trigger the browser instances with tasks. The browser instances are running as a jar file in glassfish V3 I definitely did not find this an easy task.... but its fun. Jason On Dec 15, 2009, at 12:00 AM, Markus Kohler wrote: > Hi all, > Does someone have simple example of a multi threaded test? > > > Regards, > Markus > _______________________________________________ > Celerity-users mailing list > Celerity-users at rubyforge.org > http://rubyforge.org/mailman/listinfo/celerity-users From markus.kohler at gmail.com Mon Dec 14 19:12:20 2009 From: markus.kohler at gmail.com (Markus Kohler) Date: Tue, 15 Dec 2009 01:12:20 +0100 Subject: [Celerity-users] Parallel running test? In-Reply-To: <771905290912141608u47218391tcc4ec44f123690d4@mail.gmail.com> References: <771905290912141500v12088bcap2aa580641516916c@mail.gmail.com> <54036DAA-C49A-4E07-9FDC-5F42F4D95EF7@googlemail.com> <771905290912141608u47218391tcc4ec44f123690d4@mail.gmail.com> Message-ID: <771905290912141612y5fb6596ana02ae87c466f8ae2@mail.gmail.com> Hi, Thanks! I tried to just spawn threads from jruby but that didn't really work properly. What kind of changes where necessary to htmlunit? Isn't it thread safe? Markus On Dec 15, 2009 12:44 AM, "jason franklin-stokes" < jasoninclass at googlemail.com> wrote: hmm, I have 100 running in Parallel.... I can't say that I can show you how that works in a few lines of code. I needed to make some changes to htmlunit and to celerity to getting running properly (and I am still not too sure that is). I ended up using rails in multithreded mode to encapsulate the browser instances and a seperate server to trigger the browser instances with tasks. The browser instances are running as a jar file in glassfish V3 I definitely did not find this an easy task.... but its fun. Jason On Dec 15, 2009, at 12:00 AM, Markus Kohler wrote: > Hi all, > Does someone have simple example o... > _______________________________________________ > Celerity-users mailing list > Celerity-users at rubyforge.org > http://rubyforge.org/mailman/listinfo/celerity-users _______________________________________________ Celerity-users mailing list Celerity-users at rubyforge.org http://rubyforge.org/mailman/listinfo/celerity-users -------------- next part -------------- An HTML attachment was scrubbed... URL: From jari.bakken at gmail.com Mon Dec 14 19:50:42 2009 From: jari.bakken at gmail.com (Jari Bakken) Date: Tue, 15 Dec 2009 01:50:42 +0100 Subject: [Celerity-users] Parallel running test? In-Reply-To: <771905290912141612y5fb6596ana02ae87c466f8ae2@mail.gmail.com> References: <771905290912141500v12088bcap2aa580641516916c@mail.gmail.com> <54036DAA-C49A-4E07-9FDC-5F42F4D95EF7@googlemail.com> <771905290912141608u47218391tcc4ec44f123690d4@mail.gmail.com> <771905290912141612y5fb6596ana02ae87c466f8ae2@mail.gmail.com> Message-ID: On Tue, Dec 15, 2009 at 1:12 AM, Markus Kohler wrote: > Hi, > Thanks! > I tried to just spawn threads from jruby but that didn't really work > properly. > What kind of changes where necessary to htmlunit? Isn't it thread safe? > Markus > What didn't work properly? HtmlUnit should be thread-safe as long as you have one browser instance per thread. Here's a simple example: http://gist.github.com/256610 From dorikick at gmail.com Tue Dec 15 00:33:35 2009 From: dorikick at gmail.com (doridori Jo) Date: Mon, 14 Dec 2009 21:33:35 -0800 Subject: [Celerity-users] Parallel running test? In-Reply-To: References: <771905290912141500v12088bcap2aa580641516916c@mail.gmail.com> <54036DAA-C49A-4E07-9FDC-5F42F4D95EF7@googlemail.com> <771905290912141608u47218391tcc4ec44f123690d4@mail.gmail.com> <771905290912141612y5fb6596ana02ae87c466f8ae2@mail.gmail.com> Message-ID: <3a8841390912142133s1b0762del3936c950b42ff40f@mail.gmail.com> any chancee you can write up a quick tutorial on that jason ? it'd be greatly appreciated.... thank you. On Mon, Dec 14, 2009 at 4:50 PM, Jari Bakken wrote: > On Tue, Dec 15, 2009 at 1:12 AM, Markus Kohler > wrote: > > Hi, > > Thanks! > > I tried to just spawn threads from jruby but that didn't really work > > properly. > > What kind of changes where necessary to htmlunit? Isn't it thread safe? > > Markus > > > > What didn't work properly? HtmlUnit should be thread-safe as long as > you have one browser instance per thread. > > Here's a simple example: > > http://gist.github.com/256610 > _______________________________________________ > Celerity-users mailing list > Celerity-users at rubyforge.org > http://rubyforge.org/mailman/listinfo/celerity-users > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jasoninclass at googlemail.com Tue Dec 15 02:40:57 2009 From: jasoninclass at googlemail.com (jason franklin-stokes) Date: Tue, 15 Dec 2009 08:40:57 +0100 Subject: [Celerity-users] Parallel running test? In-Reply-To: <771905290912141612y5fb6596ana02ae87c466f8ae2@mail.gmail.com> References: <771905290912141500v12088bcap2aa580641516916c@mail.gmail.com> <54036DAA-C49A-4E07-9FDC-5F42F4D95EF7@googlemail.com> <771905290912141608u47218391tcc4ec44f123690d4@mail.gmail.com> <771905290912141612y5fb6596ana02ae87c466f8ae2@mail.gmail.com> Message-ID: <8092A631-DD16-4D61-8997-10CF9ED528F0@googlemail.com> Markus, just to clarify - htmlunit is thread safe!! no doubts about that - but its not just about htmlunit - its going to be a combi of your code and htmlunit - and the resources you have available for your goals and the strategy you go for. I am spawning a lot of instances that go out doing their own thing in unknown territory - as quickly as possible -using as little new code as possible - which is what I am assuming what you wanna be doing - as you are looking at multithreading. At the end of the day it is mostly about your htmlunit error handling and thread stratgey/recovery/monitoring and resource management. I personally did not find this trivial. I am planning to blog the details in after xmas - but for now here is a shopping list: - atomize the tasks you want htmlunit to do - i.e. no not use huge loops that do a ton of things. - use rails to encapsulate your htmlunit tasks - trigger the tasks (i use restclient) that sends the task information to a rails controller that in turn triggers your htmlunit tasks. - put rails in multithread mode so that you trigger the htmlunit in parralell - launch the app in some application server (i use glassfish V3) - monitor the threads and memory useage accordingly (I use webvm for debugging and monitoring). - always always always close browser when you are finished with an instance. - I needed to add a method to celerity in class browser and class clickable element that forrces a page to open in a new window when you click it - it has the same effect as click_and_attach but without the overhead (i could not get click_and_attach working properly for what i needed in my environment - but this might not apply to you - it depends what you want to do) It might seem a little strange to encapsulate htmlunit in the rails framework - but rails is thread safe and so is htmlunit - and you have database persistance at hand if you need it - all you have to do is put it together, rather reinventing the wheel - which means a lot less can go wrong - and its not that much additional overhead. You can expect to need between 30MB and 100MB per browser instance depending on the website and the tasks it performs etc - and the time needed before GC kicks in for you to clean up the mess after you. P.S the other changes I made to htmlunit and celerity all have to do with persistance in a multithreading environment and less with mutithreading itself. I hope that makes some sence and is of some help. Jason. On Dec 15, 2009, at 1:12 AM, Markus Kohler wrote: > Hi, > Thanks! > I tried to just spawn threads from jruby but that didn't really work properly. > What kind of changes where necessary to htmlunit? Isn't it thread safe? > Markus > > >> On Dec 15, 2009 12:44 AM, "jason franklin-stokes" wrote: >> >> hmm, >> >> I have 100 running in Parallel.... >> >> I can't say that I can show you how that works in a few lines of code. >> >> I needed to make some changes to htmlunit and to celerity to getting running properly (and I am still not too sure that is). >> >> I ended up using rails in multithreded mode to encapsulate the browser instances and a seperate server to trigger the browser instances with tasks. >> >> The browser instances are running as a jar file in glassfish V3 >> >> I definitely did not find this an easy task.... but its fun. >> >> Jason >> On Dec 15, 2009, at 12:00 AM, Markus Kohler wrote: > Hi all, > Does someone have simple example o... >> >> > _______________________________________________ >> > Celerity-users mailing list >> > Celerity-users at rubyforge.org >> > http://rubyforge.org/mailman/listinfo/celerity-users >> >> _______________________________________________ >> Celerity-users mailing list >> Celerity-users at rubyforge.org >> http://rubyforge.org/mailman/listinfo/celerity-users > > _______________________________________________ > Celerity-users mailing list > Celerity-users at rubyforge.org > http://rubyforge.org/mailman/listinfo/celerity-users -------------- next part -------------- An HTML attachment was scrubbed... URL: From markus.kohler at gmail.com Tue Dec 15 03:05:52 2009 From: markus.kohler at gmail.com (Markus Kohler) Date: Tue, 15 Dec 2009 09:05:52 +0100 Subject: [Celerity-users] Parallel running test? In-Reply-To: <8092A631-DD16-4D61-8997-10CF9ED528F0@googlemail.com> References: <771905290912141500v12088bcap2aa580641516916c@mail.gmail.com> <54036DAA-C49A-4E07-9FDC-5F42F4D95EF7@googlemail.com> <771905290912141608u47218391tcc4ec44f123690d4@mail.gmail.com> <771905290912141612y5fb6596ana02ae87c466f8ae2@mail.gmail.com> <8092A631-DD16-4D61-8997-10CF9ED528F0@googlemail.com> Message-ID: <771905290912150005u30b09e1dq5d39fd58f7246403@mail.gmail.com> Hi Jason, Thanks a lot for your reply! Makes a lot of sense to me. I'm looking forward to your blog post, where should I look for it? :-) BTW your link on twitter is broken. Markus "The best way to predict the future is to invent it" -- Alan Kay On Tue, Dec 15, 2009 at 8:40 AM, jason franklin-stokes < jasoninclass at googlemail.com> wrote: > > Markus, > > just to clarify - htmlunit is thread safe!! no doubts about that - but its > not just about htmlunit - its going to be a combi of your code and htmlunit > - and the resources you have available for your goals and the strategy you > go for. I am spawning a lot of instances that go out doing their own thing > in unknown territory - as quickly as possible -using as little new code as > possible - which is what I am assuming what you wanna be doing - as you are > looking at multithreading. At the end of the day it is mostly about your > htmlunit error handling and thread stratgey/recovery/monitoring and resource > management. I personally did not find this trivial. > > I am planning to blog the details in after xmas - but for now here is a > shopping list: > > - atomize the tasks you want htmlunit to do - i.e. no not use huge loops > that do a ton of things. > - use rails to encapsulate your htmlunit tasks > - trigger the tasks (i use restclient) that sends the task information to a > rails controller that in turn triggers your htmlunit tasks. > - put rails in multithread mode so that you trigger the htmlunit in > parralell > - launch the app in some application server (i use glassfish V3) > - monitor the threads and memory useage accordingly (I use webvm for > debugging and monitoring). > - always always always close browser when you are finished with an > instance. > - I needed to add a method to celerity in class browser and class clickable > element that forrces a page to open in a new window when you click it - it > has the same effect as click_and_attach but without the overhead (i could > not get click_and_attach working properly for what i needed in my > environment - but this might not apply to you - it depends what you want to > do) > > It might seem a little strange to encapsulate htmlunit in the rails > framework - but rails is thread safe and so is htmlunit - and you have > database persistance at hand if you need it - all you have to do is put it > together, rather reinventing the wheel - which means a lot less can go wrong > - and its not that much additional overhead. > > You can expect to need between 30MB and 100MB per browser instance > depending on the website and the tasks it performs etc - and the time needed > before GC kicks in for you to clean up the mess after you. > > P.S the other changes I made to htmlunit and celerity all have to do with > persistance in a multithreading environment and less with mutithreading > itself. > > I hope that makes some sence and is of some help. > > Jason. > > > > On Dec 15, 2009, at 1:12 AM, Markus Kohler wrote: > > Hi, > Thanks! > I tried to just spawn threads from jruby but that didn't really work > properly. > What kind of changes where necessary to htmlunit? Isn't it thread safe? > Markus > > On Dec 15, 2009 12:44 AM, "jason franklin-stokes" < > jasoninclass at googlemail.com> wrote: > > hmm, > > I have 100 running in Parallel.... > > I can't say that I can show you how that works in a few lines of code. > > I needed to make some changes to htmlunit and to celerity to getting > running properly (and I am still not too sure that is). > > I ended up using rails in multithreded mode to encapsulate the browser > instances and a seperate server to trigger the browser instances with tasks. > > The browser instances are running as a jar file in glassfish V3 > > I definitely did not find this an easy task.... but its fun. > > Jason > > On Dec 15, 2009, at 12:00 AM, Markus Kohler wrote: > Hi all, > Does someone > have simple example o... > > _______________________________________________ > > Celerity-users mailing list > > Celerity-users at rubyforge.org > > http://rubyforge.org/mailman/listinfo/celerity-users > > _______________________________________________ > Celerity-users mailing list > Celerity-users at rubyforge.org > http://rubyforge.org/mailman/listinfo/celerity-users > > > _______________________________________________ > Celerity-users mailing list > Celerity-users at rubyforge.org > http://rubyforge.org/mailman/listinfo/celerity-users > > > > _______________________________________________ > Celerity-users mailing list > Celerity-users at rubyforge.org > http://rubyforge.org/mailman/listinfo/celerity-users > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From markus.kohler at gmail.com Tue Dec 15 03:33:14 2009 From: markus.kohler at gmail.com (Markus Kohler) Date: Tue, 15 Dec 2009 09:33:14 +0100 Subject: [Celerity-users] Parallel running test? In-Reply-To: References: <771905290912141500v12088bcap2aa580641516916c@mail.gmail.com> <54036DAA-C49A-4E07-9FDC-5F42F4D95EF7@googlemail.com> <771905290912141608u47218391tcc4ec44f123690d4@mail.gmail.com> <771905290912141612y5fb6596ana02ae87c466f8ae2@mail.gmail.com> Message-ID: <771905290912150033k464fc1bcme763f9e21020fc4f@mail.gmail.com> Thanks! I will check that example. Markus "The best way to predict the future is to invent it" -- Alan Kay On Tue, Dec 15, 2009 at 1:50 AM, Jari Bakken wrote: > On Tue, Dec 15, 2009 at 1:12 AM, Markus Kohler > wrote: > > Hi, > > Thanks! > > I tried to just spawn threads from jruby but that didn't really work > > properly. > > What kind of changes where necessary to htmlunit? Isn't it thread safe? > > Markus > > > > What didn't work properly? HtmlUnit should be thread-safe as long as > you have one browser instance per thread. > > Here's a simple example: > > http://gist.github.com/256610 > _______________________________________________ > Celerity-users mailing list > Celerity-users at rubyforge.org > http://rubyforge.org/mailman/listinfo/celerity-users > -------------- next part -------------- An HTML attachment was scrubbed... URL: From markus.kohler at gmail.com Tue Dec 15 04:23:13 2009 From: markus.kohler at gmail.com (Markus Kohler) Date: Tue, 15 Dec 2009 10:23:13 +0100 Subject: [Celerity-users] Parallel running test? In-Reply-To: References: <771905290912141500v12088bcap2aa580641516916c@mail.gmail.com> <54036DAA-C49A-4E07-9FDC-5F42F4D95EF7@googlemail.com> <771905290912141608u47218391tcc4ec44f123690d4@mail.gmail.com> <771905290912141612y5fb6596ana02ae87c466f8ae2@mail.gmail.com> Message-ID: <771905290912150123r6249b0c8ha69263cc167156cc@mail.gmail.com> Hi, Thanks It seems I found the issue. Do to my lack of knowledge about how threading works in Ruby I used the wrong kind of loop. E.g. I used a "for i in" loop which didn't work, but n.times does work. Regards, Markus "The best way to predict the future is to invent it" -- Alan Kay On Tue, Dec 15, 2009 at 1:50 AM, Jari Bakken wrote: > On Tue, Dec 15, 2009 at 1:12 AM, Markus Kohler > wrote: > > Hi, > > Thanks! > > I tried to just spawn threads from jruby but that didn't really work > > properly. > > What kind of changes where necessary to htmlunit? Isn't it thread safe? > > Markus > > > > What didn't work properly? HtmlUnit should be thread-safe as long as > you have one browser instance per thread. > > Here's a simple example: > > http://gist.github.com/256610 > _______________________________________________ > Celerity-users mailing list > Celerity-users at rubyforge.org > http://rubyforge.org/mailman/listinfo/celerity-users > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jasoninclass at googlemail.com Tue Dec 15 05:06:06 2009 From: jasoninclass at googlemail.com (jason franklin-stokes) Date: Tue, 15 Dec 2009 11:06:06 +0100 Subject: [Celerity-users] Parallel running test? In-Reply-To: <771905290912150005u30b09e1dq5d39fd58f7246403@mail.gmail.com> References: <771905290912141500v12088bcap2aa580641516916c@mail.gmail.com> <54036DAA-C49A-4E07-9FDC-5F42F4D95EF7@googlemail.com> <771905290912141608u47218391tcc4ec44f123690d4@mail.gmail.com> <771905290912141612y5fb6596ana02ae87c466f8ae2@mail.gmail.com> <8092A631-DD16-4D61-8997-10CF9ED528F0@googlemail.com> <771905290912150005u30b09e1dq5d39fd58f7246403@mail.gmail.com> Message-ID: <44B8DC0F-D517-4C99-BC3F-53979AB228EF@googlemail.com> http://twitter.com/jasonFS not very informative http://jasonsrailsblog.blogspot.com/ #just my note book - which i last used last year - tut tut. On Dec 15, 2009, at 9:05 AM, Markus Kohler wrote: > kes some sence and is of From markus.kohler at gmail.com Tue Dec 15 05:08:57 2009 From: markus.kohler at gmail.com (Markus Kohler) Date: Tue, 15 Dec 2009 11:08:57 +0100 Subject: [Celerity-users] Parallel running test? In-Reply-To: <44B8DC0F-D517-4C99-BC3F-53979AB228EF@googlemail.com> References: <771905290912141500v12088bcap2aa580641516916c@mail.gmail.com> <54036DAA-C49A-4E07-9FDC-5F42F4D95EF7@googlemail.com> <771905290912141608u47218391tcc4ec44f123690d4@mail.gmail.com> <771905290912141612y5fb6596ana02ae87c466f8ae2@mail.gmail.com> <8092A631-DD16-4D61-8997-10CF9ED528F0@googlemail.com> <771905290912150005u30b09e1dq5d39fd58f7246403@mail.gmail.com> <44B8DC0F-D517-4C99-BC3F-53979AB228EF@googlemail.com> Message-ID: <771905290912150208h3b051390x5a51cb9aa0dd9ab2@mail.gmail.com> Thanks! Markus "The best way to predict the future is to invent it" -- Alan Kay On Tue, Dec 15, 2009 at 11:06 AM, jason franklin-stokes < jasoninclass at googlemail.com> wrote: > > > http://twitter.com/jasonFS > not very informative > > http://jasonsrailsblog.blogspot.com/ > #just my note book - which i last used last year - tut tut. > > > On Dec 15, 2009, at 9:05 AM, Markus Kohler wrote: > > > kes some sence and is of > > _______________________________________________ > Celerity-users mailing list > Celerity-users at rubyforge.org > http://rubyforge.org/mailman/listinfo/celerity-users > -------------- next part -------------- An HTML attachment was scrubbed... URL: From dorikick at gmail.com Tue Dec 15 14:18:01 2009 From: dorikick at gmail.com (doridori Jo) Date: Tue, 15 Dec 2009 11:18:01 -0800 Subject: [Celerity-users] Parallel running test? In-Reply-To: <771905290912150208h3b051390x5a51cb9aa0dd9ab2@mail.gmail.com> References: <771905290912141500v12088bcap2aa580641516916c@mail.gmail.com> <54036DAA-C49A-4E07-9FDC-5F42F4D95EF7@googlemail.com> <771905290912141608u47218391tcc4ec44f123690d4@mail.gmail.com> <771905290912141612y5fb6596ana02ae87c466f8ae2@mail.gmail.com> <8092A631-DD16-4D61-8997-10CF9ED528F0@googlemail.com> <771905290912150005u30b09e1dq5d39fd58f7246403@mail.gmail.com> <44B8DC0F-D517-4C99-BC3F-53979AB228EF@googlemail.com> <771905290912150208h3b051390x5a51cb9aa0dd9ab2@mail.gmail.com> Message-ID: <3a8841390912151118s6aa50ac2nc0b9bc5ca9e3d16b@mail.gmail.com> Thanks Jason ! On Tue, Dec 15, 2009 at 2:08 AM, Markus Kohler wrote: > Thanks! > Markus > "The best way to predict the future is to invent it" -- Alan Kay > > > On Tue, Dec 15, 2009 at 11:06 AM, jason franklin-stokes < > jasoninclass at googlemail.com> wrote: > >> >> >> http://twitter.com/jasonFS >> not very informative >> >> http://jasonsrailsblog.blogspot.com/ >> #just my note book - which i >> last used last year - tut tut. >> >> >> On Dec 15, 2009, at 9:05 AM, Markus Kohler wrote: >> >> > kes some sence and is of >> >> _______________________________________________ >> Celerity-users mailing list >> Celerity-users at rubyforge.org >> http://rubyforge.org/mailman/listinfo/celerity-users >> > > > _______________________________________________ > Celerity-users mailing list > Celerity-users at rubyforge.org > http://rubyforge.org/mailman/listinfo/celerity-users > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jasoninclass at googlemail.com Thu Dec 17 13:49:03 2009 From: jasoninclass at googlemail.com (jason franklin-stokes) Date: Thu, 17 Dec 2009 19:49:03 +0100 Subject: [Celerity-users] celerity war file https - page not loaded Message-ID: <2C295BB9-E9B2-45AB-9ECF-194B4A2EF015@googlemail.com> Dear all, I am having a slight issue here. I have my project packed in a war file deploying from glassfish (NOTE this does not happen when just starting the app form the command line with Jruby -S myapp). I am always getting a page not loaded exceptiion with the following: browser = Celerity::Browser.new browser.goto "https://secure.wittich.de/index.php?id=170" browser.url the glassfish server log file reports this error: org.jruby.rack.RackInitializationException: java.net.SocketException: java.security.NoSuchAlgorithmException: Error constructing implementation (algorithm: Default, provider: SunJSSE, class: com.sun.net.ssl.internal.ssl.DefaultSSLContextImpl) from javax/net/ssl/SSLSocketFactory.java:212:in `createSocket' from org/apache/commons/httpclient/protocol/SSLProtocolSocketFactory.java:82:in `createSocket' from org/apache/commons/httpclient/protocol/SSLProtocolSocketFactory.java:127:in `createSocket' from org/apache/commons/httpclient/HttpConnection.java:707:in `open' from org/apache/commons/httpclient/MultiThreadedHttpConnectionManager.java:1361:in `open' from org/apache/commons/httpclient/HttpMethodDirector.java:387:in `executeWithRetry' from org/apache/commons/httpclient/HttpMethodDirector.java:171:in `executeMethod' from org/apache/commons/httpclient/HttpClient.java:397:in `executeMethod' I have the jruby-openssl gem which is being required correctly. Has anyone got any great ideas on how I could resolve this. Thanks a million. Jason. From jari.bakken at gmail.com Thu Dec 17 15:09:22 2009 From: jari.bakken at gmail.com (Jari Bakken) Date: Thu, 17 Dec 2009 21:09:22 +0100 Subject: [Celerity-users] celerity war file https - page not loaded In-Reply-To: <2C295BB9-E9B2-45AB-9ECF-194B4A2EF015@googlemail.com> References: <2C295BB9-E9B2-45AB-9ECF-194B4A2EF015@googlemail.com> Message-ID: On Thu, Dec 17, 2009 at 7:49 PM, jason franklin-stokes wrote: > > org.jruby.rack.RackInitializationException: java.net.SocketException: java.security.NoSuchAlgorithmException: Error constructing implementation (algorithm: Default, provider: SunJSSE, class: com.sun.net.ssl.internal.ssl.DefaultSSLContextImpl) > ? ? ? ?from javax/net/ssl/SSLSocketFactory.java:212:in `createSocket' > ? ? ? ?from org/apache/commons/httpclient/protocol/SSLProtocolSocketFactory.java:82:in `createSocket' > ? ? ? ?from org/apache/commons/httpclient/protocol/SSLProtocolSocketFactory.java:127:in `createSocket' > ? ? ? ?from org/apache/commons/httpclient/HttpConnection.java:707:in `open' > ? ? ? ?from org/apache/commons/httpclient/MultiThreadedHttpConnectionManager.java:1361:in `open' > ? ? ? ?from org/apache/commons/httpclient/HttpMethodDirector.java:387:in `executeWithRetry' > ? ? ? ?from org/apache/commons/httpclient/HttpMethodDirector.java:171:in `executeMethod' > ? ? ? ?from org/apache/commons/httpclient/HttpClient.java:397:in `executeMethod' > > I have the jruby-openssl gem which is being required correctly. > > Has anyone got any great ideas on how I could resolve this. > You could try turning off secure ssl: Celerity::Browser.new(:secure_ssl => false) Otherwise I think this is one for the HtmlUnit list. From dorikick at gmail.com Wed Dec 30 03:51:21 2009 From: dorikick at gmail.com (doridori Jo) Date: Wed, 30 Dec 2009 00:51:21 -0800 Subject: [Celerity-users] possible to test flash applications ? Message-ID: <3a8841390912300051k3b9566c1xe68aec01ad4651ad@mail.gmail.com> is it possible for celerity to click on say flash buttons ? -------------- next part -------------- An HTML attachment was scrubbed... URL: From jari.bakken at gmail.com Wed Dec 30 06:52:27 2009 From: jari.bakken at gmail.com (Jari Bakken) Date: Wed, 30 Dec 2009 12:52:27 +0100 Subject: [Celerity-users] possible to test flash applications ? In-Reply-To: <3a8841390912300051k3b9566c1xe68aec01ad4651ad@mail.gmail.com> References: <3a8841390912300051k3b9566c1xe68aec01ad4651ad@mail.gmail.com> Message-ID: On Wed, Dec 30, 2009 at 9:51 AM, doridori Jo wrote: > is it possible for celerity to click on say flash buttons ? > No, Flash is not currently supported by HtmlUnit. You might want to look at one of these tools: http://funfx.rubyforge.org/ http://code.google.com/p/flash-watir/ http://code.google.com/p/flash-selenium/