' > Only change I made was to swap File.exists? (that is deprecated) by File.exist? at lib/mongrel/logger.rb and compile mongrel through setup.rb only. As I already commited this change to svn right now, the ones who update it may have a clean 1.9 run. > > So, now that rails run on top of it, i'll start to test test test test test test and test. > I think I described the situation: more users will start firing huge projects into 1.9 and mostly they will blame Mongrel for all of them (as usual, if not, check mongrel-users list). > If we drop support requirement for fastthread in mongrel 1.2, mongrel will be 1.9 compatible out-of-the-box. > Is not that simple, but anyway... be my guess, try it ourself and shoot you in your foot :-P -- Luis Lavena Multimedia systems - A common mistake that people make when trying to design something completely foolproof is to underestimate the ingenuity of complete fools. Douglas Adams From filipe at icewall.org Tue Dec 18 11:20:43 2007 From: filipe at icewall.org (Filipe Lautert) Date: Tue, 18 Dec 2007 14:20:43 -0200 Subject: [Mongrel-development] First Shoot, many more to appear: Rails on Ruby 1.9 In-Reply-To: <71166b3b0712180624p3a6e8025mc0cf0b50e291b32e@mail.gmail.com> References: <71166b3b0712172344l390ff828wcbb8ace7a0f4c118@mail.gmail.com> <71166b3b0712180624p3a6e8025mc0cf0b50e291b32e@mail.gmail.com> Message-ID: <1197994843.7533.6.camel@renmazuo> On Ter, 2007-12-18 at 11:24 -0300, Luis Lavena wrote: > On Dec 18, 2007 9:42 AM, Filipe wrote: > > ... > > Tad?, and mongrel is running and serving requests with ruby1.9 and rails 2.0.2 trunk :) > > > > Yeah, running doesn't mean it will work as good as 1.8. For sure :) What I want at first is to make mongrel ruby1.9 compatible. If we can make it build and at least start, so wee can get more people to test it. You know, we can do a lot of tests but we will never fire all the bugs that are hidden.. so we can release early and got help from users. What we can do is advise users that mongrel + ruby1.9 is a dangerous combination, and that compatible != stable. Cheers, filipe From wyhaines at gmail.com Tue Dec 18 12:27:32 2007 From: wyhaines at gmail.com (Kirk Haines) Date: Tue, 18 Dec 2007 10:27:32 -0700 Subject: [Mongrel-development] The case against EM (for now) (was: Some silly benchs) In-Reply-To: References: <71166b3b0712141143k44808f7am357c0c2573c23565@mail.gmail.com> <20071214203732.c1a9e576.zedshaw@zedshaw.com> <20071216114550.3db98415.zedshaw@zedshaw.com> <20071216160852.a49e2ebf.zedshaw@zedshaw.com> <20071218054652.dcc3a595.zedshaw@zedshaw.com> Message-ID: On Dec 18, 2007 3:46 AM, Zed A. Shaw wrote: > On Sun, 16 Dec 2007 17:00:38 -0700 > "Kirk Haines" wrote: > > > In order for EM to do what couldn't be done with Ruby/Event, it has to > > coexist with the Ruby threads. > > EmSelect is ruby_thread_select. > > And that means it's *still* using select. You haven't disproven this, > and it's a shitty design. LOL. No, that means that it is written so that it works with Ruby's design, to permit Ruby's threads to run without deadlocking. Specifically, EM's epoll loop processes epoll events, and then it runs rb_thread_select with no descriptors in order to give Ruby's threads a chance to run. It's a good design for the requirements, which involve being about to work with Ruby's threading mechanism. > However, the statements you made, and other make about EM are that it > somehow sidesteps this. That's a lie, and one more reason not to use > the project. If the statements were more honest like, "it layers an > epoll system on top of ruby's select so not super fast as epol, but > getting better" then it might be excusable. However, instead it's > hyperbole about using epoll, and it's been this way forever. *I* have tested my apps with up to 20000 open, active simultaneous connections on Linux platforms. You're calling me a liar. Don't do that. As for your other assertion, EM supports Ruby thread interoperability because it has to. It's as simple as that. Your own experience with Ruby/Event's failure demonstrates why it has to. > The project has claimed Ruby licensed for years and didn't change the > license statements in source until I mentioned it. You even missed > that, which you constantly seem to do which is why I don't trust your > statements about even basic assertions. Again, you go on with the unsupported statements that I missed things. I didn't. EventMachine has existed as a project since April of 2006 (i.e. less than 2 years), and it has always been dual licensed. In August of 2006 you pointed out some incorrect source file headings, and they were fixed. > > It coexists with rb_thread_select. That's the crucial missing part of > > your analysis. > > > > EM is using epoll, but EM does its work by interleaving with Ruby's > > select based thread management so that ruby threads outside of EM have > > a chance to run, without deadlocks. > > And again Kirk, that's a lie. It does not "interleave with > Ruby's select" which implies that the epoll descriptors are handled off > to the side and are not impacted by Ruby's select or that Ruby's > select isn't involved in the processing. If what you say is true, why is > this code in em.cpp: > > // prepare the sockets for reading and writing > size_t i; > for (i = 0; i < Descriptors.size(); i++) { > EventableDescriptor *ed = Descriptors[i]; > assert (ed); > int sd = ed->GetSocket(); > assert (sd != INVALID_SOCKET); > > if (ed->SelectForRead()) > FD_SET (sd, &fdreads); > if (ed->SelectForWrite()) > FD_SET (sd, &fdwrites); > > if (maxsocket < sd) > maxsocket = sd; > } > > THAT is loading the file descriptors into a god damned read/write set > so they can be processed by Ruby. THAT means that Ruby's select > is still processing them. THAT means you are still limited by > Ruby's IO, GC, and thread limitations. > > Now Kirk, how is it that you missed this? I know you're not the kind > of guy to just bold-faced lie about something, so is it that you're > having a hard time reading the C++ code? Obviously all that code is a > problem then. I didn't miss anythiing, here, Zed, nor did I lie. Don't misunderstand me, here. Desist from that libelous assertion. I don't appreciate your defamation. /************************ EventMachine_t::_RunOnce ************************/ bool EventMachine_t::_RunOnce() { if (bEpoll) return _RunEpollOnce(); else return _RunSelectOnce(); } This dispatches to the appropriate handler depending on whether epoll is being used or not. The code that you quoted is from _RunSelectOnce(). As you can see above, that code is used only if one is using select. If one is using epoll, _RunEpollOnce() is called; select() isn't used. Your assertion is incorrect. And again, to be quite clear, quit calling me a liar. I did not misunderstand the code. You very clearly did. And *I* have not intentionally misrepresented anything. > > And *I* am happy to put time towards addressing real issues. > > I'm guessing you mean nothing I've brought up with the 10-20 minutes of > effort nobody else seems to want to exert on the project before using > it. No. It means that I am right here, listening to you, and am happy to look at the issues and see if any of them are real problems. That is, I will look at them and analyze them instead of just taking your word of authority on the matter. If my analysis agrees with yours, then I will fix them. > What the fuck, so you tell me that I'm vitriolic. I show that a SINGLE > DAMN GREP finds massive security holes like loading a buffer in a loop > via sprintf (binder.cpp) or the use of a clear text password in the > source as the basis of my assertion. You don't say, "Damn Zed, you were > right, that does suck." You say, "Taking note, thanks man." No, you showed that a single grep finds a number of things without context. One of the things that grep looks for is the use of a variable named 'index'. That's hardly a security hole. None of those lines would have showed up in the magic grep had a different variable name been used in that place. So what I am saying is that those lines deserve to be examined in context, and the use of the dangerous str* functions needs to be looked at closely, because, as you said, there usually are safer functions to use. I'm saying that there could well be room for improvement there. What I am not doing is getting aggitated, or excited, or fearful, or antagonistic. In that place you pointed out some things that need to be looked at more closely, and I will be doing so. Speaking of looking closely at things, regarding the sprintf usage in binder (which comprised a large number of the lines that your grep found): That's not a security violation. The value is being copied to an automatic array with a carefully-computed size, and the data is not coming from user input but is generated internally. > I'm guessing there's no way you'd ever admit that EM isn't that good, > just what's available, so I'll just leave this email as my last > statement on it until you start pimping it again. I haven't pimped it for Mongrel. I've _never_ said that it should be used in Mongrel core. I've resisted that notion, in fact. I just resist your defamation of EM, and of me, for using it. I would like an easier, cleaner way for things to swap out the core request handling pieces of Mongrel, but if Mongrel were mine, I'd work very hard to make sure the _core_ of Mongrel is as simple, and as base-ruby as possible. For 1.9, that means one of two things, I think. Whether this is fibers, a thread pool that plucks things off the accept queue, or some combination of the two, I don't know. On Linux and Windows, system threads per request are going to be slow. On Solaris, though, it stands a real chance of performing all right because Solaris threads are very lightweight. And since, as you have often pointed out, Mongrel is a library, and a LOT of people use it in a lot of different ways, were it mine, I'd be embracing that fact and making sure that it continues to be a flexible platform that people can adopt to their own needs. > > > Nice, much better than when I first looked at the code, but still quite > > > a few potential attack vectors. But wait a minute, what's this > > > "kittycat" in ssl.cpp? > > > > The first pass of the SSL support used hardcoded privkey and creds. > > Those are still there, but anyone with any sense uses set_tls_params() > > to provide their own privkey and cert. > > Riiiight, that's never been an issue. People *never* accidentally use > defaults for stuff. Totally secure. "kittycat" is dead code. It's a placeholder for an encryption key in case users want to add one themselves. It was originally there for decrypting the default private key. Since the private key right there in open source code for all to see, there's no point in encrypting it. Anyone who understands how X509 works would not use the default credential for authentication, but only for encryption. So in the end it comes down to doing a little RTFM and knowing what you are doing. The user can make the choices and is responsible for the outcome of those choices. > Now, you and anyone else can keep on using the damn thing--I care > less--but when you keep mentioning it for Mongrel it kind of pisses me > off. Do what you want outside Mongrel, but all of the above reasons > (and then some) make me not want to have it within 10 feet of the > Mongrel project. As I mentioned above, I have never advocated using it for Mongrel itself. I have said as much to Evan, Wayne, Luis, and Fillipe during IRC conversations. I have only advocated for EM itself in this forum because you persist with the shouting, name calling, and inaccuracies in your statements. > So Please Kirk, quit mentioning it, quit mentioning that it is "faster > than Ruby's select" which is a lie, and quit saying that my objections > to it are based on anything other than the facts I find right in the > source or online. It's not vitriol or opinion, it's based on a review > of the source that you obviously aren't doing. /************************ EventMachine_t::_RunOnce ************************/ bool EventMachine_t::_RunOnce() { if (bEpoll) return _RunEpollOnce(); else return _RunSelectOnce(); } Kirk Haines From luislavena at gmail.com Tue Dec 25 17:58:15 2007 From: luislavena at gmail.com (Luis Lavena) Date: Tue, 25 Dec 2007 19:58:15 -0300 Subject: [Mongrel-development] Review of Code for 1.9 Message-ID: <71166b3b0712251458h2227853ew2a8d652a81ce0706@mail.gmail.com> Hello Guys, I'm reviewing the code for 1.9, and forgot about this when we first spoke on this subject. The current way we stop threads is using Thread#raise to spread StopServer exception, which will not work as expected in 1.9. 1.9 will treat raised exceptions as #kill, like JRuby does, so the worker threads will not finish serving the client and _then_ exiting, but will be kill'ed in the middle of the request. (at the bottom of mongrel.rb): http://mongrel.rubyforge.org/svn/trunk/lib/mongrel.rb According to some docs of Jruby and YARV, we should rely on 'thread safety' of these methods (including #critical and #kill). Thoughts? btw, merry christmas to everybody! :-D -- Luis Lavena Multimedia systems - A common mistake that people make when trying to design something completely foolproof is to underestimate the ingenuity of complete fools. Douglas Adams From filipe at icewall.org Tue Dec 25 19:05:46 2007 From: filipe at icewall.org (Filipe) Date: Tue, 25 Dec 2007 22:05:46 -0200 (BRST) Subject: [Mongrel-development] Review of Code for 1.9 In-Reply-To: <71166b3b0712251458h2227853ew2a8d652a81ce0706@mail.gmail.com> References: <71166b3b0712251458h2227853ew2a8d652a81ce0706@mail.gmail.com> Message-ID: On Tue, 25 Dec 2007, Luis Lavena wrote: > Hello Guys, > > I'm reviewing the code for 1.9, and forgot about this when we first > spoke on this subject. > > The current way we stop threads is using Thread#raise to spread > StopServer exception, which will not work as expected in 1.9. > > 1.9 will treat raised exceptions as #kill, like JRuby does, so the > worker threads will not finish serving the client and _then_ exiting, > but will be kill'ed in the middle of the request. > > (at the bottom of mongrel.rb): > http://mongrel.rubyforge.org/svn/trunk/lib/mongrel.rb > > According to some docs of Jruby and YARV, we should rely on 'thread > safety' of these methods (including #critical and #kill). Hum.. could you point where are those docs? Easier ask than search, as you already know them.... Anyway, we can still raising an Exception to it, just to stop the accept thing. But then we can call graceful_shutdown method within stop, and not within the @acceptor thread. Something like this: --- lib/mongrel.rb (revision 919) +++ lib/mongrel.rb (working copy) @@ -307,9 +307,6 @@ Mongrel.log(:error, e.backtrace.join("\n")) end end - graceful_shutdown - ensure - @socket.close # Mongrel.log(:error, "#{Time.now.httpdate}: Closed socket.") end end @@ -349,6 +346,9 @@ if synchronous sleep(0.5) while @acceptor.alive? end + + graceful_shutdown + @socket.close end end How about it? > > btw, merry christmas to everybody! :-D For you too! Cheers, filipe { @ icewall.org GPG 1024D/A6BA423E Jabber lautert at jabber.ru } From luislavena at gmail.com Wed Dec 26 01:46:28 2007 From: luislavena at gmail.com (Luis Lavena) Date: Wed, 26 Dec 2007 03:46:28 -0300 Subject: [Mongrel-development] Review of Code for 1.9 In-Reply-To: References: <71166b3b0712251458h2227853ew2a8d652a81ce0706@mail.gmail.com> Message-ID: <71166b3b0712252246xf3d6503s972f83c22a008ee0@mail.gmail.com> On Dec 25, 2007 9:05 PM, Filipe wrote: [...] > > > > According to some docs of Jruby and YARV, we should rely on 'thread > > safety' of these methods (including #critical and #kill). > > Hum.. could you point where are those docs? Easier ask than search, as > you already know them.... > infoQ: http://www.infoq.com/news/2007/05/ruby-threading-futures Sasada Koichi post at ruby-core: http://blade.nagaokaut.ac.jp/cgi-bin/scat.rb/ruby/ruby-core/10252 Headius, first quarter of 2007: http://headius.blogspot.com/2007_04_25_archive.html > Anyway, we can still raising an Exception to it, just to stop the accept thing. > But then we can call graceful_shutdown method within stop, and not within the @acceptor > thread. Something like this: > The problem is that raising a exception to another thread (Thread#raise) will not work as expected in different interpreters, but maybe I'm reading the docs wrong. > --- lib/mongrel.rb (revision 919) > +++ lib/mongrel.rb (working copy) > @@ -307,9 +307,6 @@ > Mongrel.log(:error, e.backtrace.join("\n")) > end > end > - graceful_shutdown > - ensure > - @socket.close > # Mongrel.log(:error, "#{Time.now.httpdate}: Closed socket.") > end > end > @@ -349,6 +346,9 @@ > if synchronous > sleep(0.5) while @acceptor.alive? > end > + > + graceful_shutdown > + @socket.close > end > > end > Mmm, need to check, no time right now. I'm bootstrapping ruby 1.9 completely from scratch for mingw (part of the new One-Click Ruby Installer). -- Luis Lavena Multimedia systems - A common mistake that people make when trying to design something completely foolproof is to underestimate the ingenuity of complete fools. Douglas Adams From zedshaw at zedshaw.com Wed Dec 26 02:26:19 2007 From: zedshaw at zedshaw.com (Zed A. Shaw) Date: Wed, 26 Dec 2007 02:26:19 -0500 Subject: [Mongrel-development] Review of Code for 1.9 In-Reply-To: <71166b3b0712251458h2227853ew2a8d652a81ce0706@mail.gmail.com> References: <71166b3b0712251458h2227853ew2a8d652a81ce0706@mail.gmail.com> Message-ID: <20071226022619.f1ab4608.zedshaw@zedshaw.com> On Tue, 25 Dec 2007 19:58:15 -0300 "Luis Lavena" wrote: > Hello Guys, > > I'm reviewing the code for 1.9, and forgot about this when we first > spoke on this subject. Don't forget to check out Rev while you're at it: http://rev.rubyforge.org/ :-) -- Zed A. Shaw - Hate: http://savingtheinternetwithhate.com/ - Good: http://www.zedshaw.com/ - Evil: http://yearofevil.com/ From luislavena at gmail.com Wed Dec 26 02:31:24 2007 From: luislavena at gmail.com (Luis Lavena) Date: Wed, 26 Dec 2007 04:31:24 -0300 Subject: [Mongrel-development] Review of Code for 1.9 In-Reply-To: <20071226022619.f1ab4608.zedshaw@zedshaw.com> References: <71166b3b0712251458h2227853ew2a8d652a81ce0706@mail.gmail.com> <20071226022619.f1ab4608.zedshaw@zedshaw.com> Message-ID: <71166b3b0712252331w2d396d3awcc44dd5b49d8148a@mail.gmail.com> On Dec 26, 2007 4:26 AM, Zed A. Shaw wrote: > On Tue, 25 Dec 2007 19:58:15 -0300 > "Luis Lavena" wrote: > > > Hello Guys, > > > > I'm reviewing the code for 1.9, and forgot about this when we first > > spoke on this subject. > > Don't forget to check out Rev while you're at it: > > http://rev.rubyforge.org/ > > :-) Didn't forgot it, but: "This includes the epoll system call for Linux, the kqueue system call for BSDs and OS X, and the completion ports interface for Solaris." No mention of Windows support or test, even that libev code works with fd and sockets on win32. I'll check that later, now I need to solve some segfaults in readline extension (for ruby 1.9 mingw32 build). Later guys, -- Luis Lavena Multimedia systems - A common mistake that people make when trying to design something completely foolproof is to underestimate the ingenuity of complete fools. Douglas Adams From wayneeseguin at gmail.com Wed Dec 26 09:33:25 2007 From: wayneeseguin at gmail.com (Wayne E. Seguin) Date: Wed, 26 Dec 2007 09:33:25 -0500 Subject: [Mongrel-development] Review of Code for 1.9 In-Reply-To: <71166b3b0712251458h2227853ew2a8d652a81ce0706@mail.gmail.com> References: <71166b3b0712251458h2227853ew2a8d652a81ce0706@mail.gmail.com> Message-ID: On Dec 25, 2007 5:58 PM, Luis Lavena wrote: > Hello Guys, > > I'm reviewing the code for 1.9, and forgot about this when we first > spoke on this subject. > > The current way we stop threads is using Thread#raise to spread > StopServer exception, which will not work as expected in 1.9. > > 1.9 will treat raised exceptions as #kill, like JRuby does, so the > worker threads will not finish serving the client and _then_ exiting, > but will be kill'ed in the middle of the request. > > (at the bottom of mongrel.rb): > http://mongrel.rubyforge.org/svn/trunk/lib/mongrel.rb > > According to some docs of Jruby and YARV, we should rely on 'thread > safety' of these methods (including #critical and #kill). > > Thoughts? > > btw, merry Christmas to everybody! :-D > -- > Luis Lavena > Multimedia systems Merry Christmas and happy holidays all!!! Perhaps we should implement a check when Mongrel is loaded which sets a variable based on platform / ruby version for which Thread method to call for stop server handling. Then at the point where we stop threads we can .send() the method stored in the variable. ~Wayne -------------- next part -------------- An HTML attachment was scrubbed... URL: http://rubyforge.org/pipermail/mongrel-development/attachments/20071226/e8669223/attachment-0001.html From luislavena at gmail.com Wed Dec 26 09:38:45 2007 From: luislavena at gmail.com (Luis Lavena) Date: Wed, 26 Dec 2007 11:38:45 -0300 Subject: [Mongrel-development] Review of Code for 1.9 In-Reply-To: References: <71166b3b0712251458h2227853ew2a8d652a81ce0706@mail.gmail.com> Message-ID: <71166b3b0712260638n6afc9899q444375674236e282@mail.gmail.com> On Dec 26, 2007 11:33 AM, Wayne E. Seguin wrote: [...] > > Merry Christmas and happy holidays all!!! > > Perhaps we should implement a check when Mongrel is loaded which sets a > variable based on platform / ruby version for which Thread method to call > for stop server handling. Then at the point where we stop threads we can > .send() the method stored in the variable. > I'll suggest that for the time being, we set required ruby version <= 1.8.6 and do some release. *everybody* is playing with 1.9 and they think is stable, they don't read the "development release" sticker matz put on it... Argh... I'm getting angry :-P -- Luis Lavena Multimedia systems - A common mistake that people make when trying to design something completely foolproof is to underestimate the ingenuity of complete fools. Douglas Adams From zedshaw at zedshaw.com Wed Dec 26 14:01:07 2007 From: zedshaw at zedshaw.com (Zed A. Shaw) Date: Wed, 26 Dec 2007 14:01:07 -0500 Subject: [Mongrel-development] Review of Code for 1.9 In-Reply-To: <71166b3b0712252331w2d396d3awcc44dd5b49d8148a@mail.gmail.com> References: <71166b3b0712251458h2227853ew2a8d652a81ce0706@mail.gmail.com> <20071226022619.f1ab4608.zedshaw@zedshaw.com> <71166b3b0712252331w2d396d3awcc44dd5b49d8148a@mail.gmail.com> Message-ID: <20071226140107.0d9aa01a.zedshaw@zedshaw.com> On Wed, 26 Dec 2007 04:31:24 -0300 "Luis Lavena" wrote: > > > > Don't forget to check out Rev while you're at it: > > > > http://rev.rubyforge.org/ > > > > :-) > > Didn't forgot it, but: > > No mention of Windows support or test, even that libev code works with > fd and sockets on win32. That's 'cause windows sucks. :-) Actually, Tony and I mostly just do unix so we haven't been motivated to work on it. Thus the reason I said you should check it out since it might help you on Windows. -- Zed A. Shaw - Hate: http://savingtheinternetwithhate.com/ - Good: http://www.zedshaw.com/ - Evil: http://yearofevil.com/ From luislavena at gmail.com Wed Dec 26 13:58:54 2007 From: luislavena at gmail.com (Luis Lavena) Date: Wed, 26 Dec 2007 15:58:54 -0300 Subject: [Mongrel-development] Review of Code for 1.9 In-Reply-To: <20071226140107.0d9aa01a.zedshaw@zedshaw.com> References: <71166b3b0712251458h2227853ew2a8d652a81ce0706@mail.gmail.com> <20071226022619.f1ab4608.zedshaw@zedshaw.com> <71166b3b0712252331w2d396d3awcc44dd5b49d8148a@mail.gmail.com> <20071226140107.0d9aa01a.zedshaw@zedshaw.com> Message-ID: <71166b3b0712261058q51e3f073md5884b46b3ecee2a@mail.gmail.com> On Dec 26, 2007 4:01 PM, Zed A. Shaw wrote: [...] > > That's 'cause windows sucks. :-) > Yeah, totally agree, but since I earn money from it... shouldn't complain too much, isn't? > Actually, Tony and I mostly just do unix so we haven't been motivated > to work on it. Thus the reason I said you should check it out since it > might help you on Windows. > Will take a look, this weekend with more spare time :-D -- Luis Lavena Multimedia systems - A common mistake that people make when trying to design something completely foolproof is to underestimate the ingenuity of complete fools. Douglas Adams From wayneeseguin at gmail.com Wed Dec 26 14:22:19 2007 From: wayneeseguin at gmail.com (Wayne E. Seguin) Date: Wed, 26 Dec 2007 14:22:19 -0500 Subject: [Mongrel-development] Review of Code for 1.9 In-Reply-To: <71166b3b0712260638n6afc9899q444375674236e282@mail.gmail.com> References: <71166b3b0712251458h2227853ew2a8d652a81ce0706@mail.gmail.com> <71166b3b0712260638n6afc9899q444375674236e282@mail.gmail.com> Message-ID: On Dec 26, 2007 9:38 AM, Luis Lavena wrote: > I'll suggest that for the time being, we set required ruby version <= > 1.8.6 and do some release. > Agreed *everybody* is playing with 1.9 and they think is stable, they don't > read the "development release" sticker matz put on it... > Perhaps branch for 1.9? ~Wayne -------------- next part -------------- An HTML attachment was scrubbed... URL: http://rubyforge.org/pipermail/mongrel-development/attachments/20071226/59766a42/attachment.html From luislavena at gmail.com Wed Dec 26 14:26:21 2007 From: luislavena at gmail.com (Luis Lavena) Date: Wed, 26 Dec 2007 16:26:21 -0300 Subject: [Mongrel-development] Review of Code for 1.9 In-Reply-To: References: <71166b3b0712251458h2227853ew2a8d652a81ce0706@mail.gmail.com> <71166b3b0712260638n6afc9899q444375674236e282@mail.gmail.com> Message-ID: <71166b3b0712261126m255d8e3di3c10f729d588938c@mail.gmail.com> On Dec 26, 2007 4:22 PM, Wayne E. Seguin wrote: > On Dec 26, 2007 9:38 AM, Luis Lavena wrote: > > > I'll suggest that for the time being, we set required ruby version <= > > 1.8.6 and do some release. > > > > Agreed > The same was commented on ruby-core: define ruby version < 1.9 and > 1.8.2 (since there could exist a 1.8.7 release). > > *everybody* is playing with 1.9 and they think is stable, they don't > > read the "development release" sticker matz put on it... > > > > Perhaps branch for 1.9? > We should leave trunk for 1.8 and bugfixing and start playing with 1.9 in a branch. We already did a lot of changes in trunk that are not 'release quality' and some experimental stuff (that shouldn't be there, for the record). Don't have enough time to start playing with some ideas (not until weekend), but feel free to branch and play safe ;-) -- Luis Lavena Multimedia systems - A common mistake that people make when trying to design something completely foolproof is to underestimate the ingenuity of complete fools. Douglas Adams From filipe at icewall.org Wed Dec 26 22:28:07 2007 From: filipe at icewall.org (Filipe) Date: Thu, 27 Dec 2007 01:28:07 -0200 (BRST) Subject: [Mongrel-development] Review of Code for 1.9 In-Reply-To: <71166b3b0712261126m255d8e3di3c10f729d588938c@mail.gmail.com> References: <71166b3b0712251458h2227853ew2a8d652a81ce0706@mail.gmail.com> <71166b3b0712260638n6afc9899q444375674236e282@mail.gmail.com> <71166b3b0712261126m255d8e3di3c10f729d588938c@mail.gmail.com> Message-ID: On Wed, 26 Dec 2007, Luis Lavena wrote: > On Dec 26, 2007 4:22 PM, Wayne E. Seguin wrote: >> On Dec 26, 2007 9:38 AM, Luis Lavena wrote: >> >>> I'll suggest that for the time being, we set required ruby version <= >>> 1.8.6 and do some release. >>> >> >> Agreed >> +1 > > The same was commented on ruby-core: > > define ruby version < 1.9 and > 1.8.2 (since there could exist a 1.8.7 release). > >>> *everybody* is playing with 1.9 and they think is stable, they don't >>> read the "development release" sticker matz put on it... >>> >> >> Perhaps branch for 1.9? >> > > We should leave trunk for 1.8 and bugfixing and start playing with 1.9 > in a branch. Hey guys, we already have a branch called stable_1-1 - evn used this to generate 1.1.2 . So this can be our stable thing and we can stay playing with trunk... > > We already did a lot of changes in trunk that are not 'release > quality' and some experimental stuff (that shouldn't be there, for the > record). Hum... not sure :) Well, we have the logger thing, but besides this I don't remember any other big change... > > Don't have enough time to start playing with some ideas (not until > weekend), but feel free to branch and play safe ;-) > Welcome back to the party! Cheers, filipe { @ icewall.org GPG 1024D/A6BA423E Jabber lautert at jabber.ru } From filipe at icewall.org Wed Dec 26 23:07:26 2007 From: filipe at icewall.org (Filipe) Date: Thu, 27 Dec 2007 02:07:26 -0200 (BRST) Subject: [Mongrel-development] Review of Code for 1.9 In-Reply-To: <71166b3b0712252246xf3d6503s972f83c22a008ee0@mail.gmail.com> References: <71166b3b0712251458h2227853ew2a8d652a81ce0706@mail.gmail.com> <71166b3b0712252246xf3d6503s972f83c22a008ee0@mail.gmail.com> Message-ID: On Wed, 26 Dec 2007, Luis Lavena wrote: > On Dec 25, 2007 9:05 PM, Filipe wrote: > [...] >>> >>> According to some docs of Jruby and YARV, we should rely on 'thread >>> safety' of these methods (including #critical and #kill). >> >> Hum.. could you point where are those docs? Easier ask than search, as >> you already know them.... >> > ... (links) Thanks for the links :) Well, one important note from ko1 in one of those links: (3') select() system call can be interrupted with pthread_kill(). But on cygwin system, pthread_kill() doesn't work well. oops. If we use the code that I sent, we may have problems wih cygwin. BUT: 1) (I hope that) no sane human will run a production server in cygwin 2) This problem would only happen in cygwin when stopping mongrel, so it's not really one of the biggest problems in the world. Also, besides using the code already sent, we could call accept_nonblock() and then get into a loop calling select in the socket for a limited time and then check a variable to exit. Something like this: serv = TCPServer.new('127.0.0.1', 3000) begin sock = serv.accept_nonblock rescue Errno::EAGAIN, Errno::ECONNABORTED, Errno::EPROTO, Errno::EINTR #executes select for 3 seconds before try our exit variable IO.select([serv], nil, nil, 3) return if die retry end This way we just set @acceptor.die = true, join it and wait. How about it? Cheers, filipe { @ icewall.org GPG 1024D/A6BA423E Jabber lautert at jabber.ru } From luislavena at gmail.com Thu Dec 27 07:01:10 2007 From: luislavena at gmail.com (Luis Lavena) Date: Thu, 27 Dec 2007 09:01:10 -0300 Subject: [Mongrel-development] Review of Code for 1.9 In-Reply-To: References: <71166b3b0712251458h2227853ew2a8d652a81ce0706@mail.gmail.com> <71166b3b0712252246xf3d6503s972f83c22a008ee0@mail.gmail.com> Message-ID: <71166b3b0712270401q68b26a69v61c57041eac387fc@mail.gmail.com> On Dec 27, 2007 1:07 AM, Filipe wrote: [...] > > Thanks for the links :) > > Well, one important note from ko1 in one of those links: > > (3') select() system call can be interrupted with pthread_kill(). But > on cygwin system, pthread_kill() doesn't work well. oops. > > If we use the code that I sent, we may have problems wih cygwin. BUT: > 1) (I hope that) no sane human will run a production server in cygwin > 2) This problem would only happen in cygwin when stopping mongrel, so it's not > really one of the biggest problems in the world. > Yeah, cygwin is not a good alternative, even that some editors (like E) encourage users to use a cygwin environment to "mimic" linux. I give support for mongrel on "mswin32" platform, not cygwin (hey, I even plan to support mingw when get it working right). > Also, besides using the code already sent, we could call accept_nonblock() > and then get into a loop calling select in the socket for a limited time and then > check a variable to exit. Something like this: > > serv = TCPServer.new('127.0.0.1', 3000) > begin > sock = serv.accept_nonblock > rescue Errno::EAGAIN, Errno::ECONNABORTED, Errno::EPROTO, Errno::EINTR > #executes select for 3 seconds before try our exit variable > IO.select([serv], nil, nil, 3) > return if die > retry > end > > This way we just set @acceptor.die = true, join it and wait. How about it? > That could work, but we will still be attached/limited/sticked/busted to Threads. one developer started a thread at ruby-core listing the 'evented' frameworks available, how bad they behave and listing part of the problems they currently present. Also, describe that something could be done in the Fibers arena to workaround this, thus removing the need for external C code: http://groups.google.com/group/ruby-core-google/browse_thread/thread/fbf8bae96b00d1f5 [ruby-core:14497] http://blade.nagaokaut.ac.jp/cgi-bin/scat.rb/ruby/ruby-core/14497 But still, don't have free time to look deep into this (damn work, I hate it, need to code some usb driver for windows by tonight). For the time being, I suggest we take stable 1.1 and: force required_ruby_version '<1.9.0' release a new gems for it (I'll do the same for the pre-build binary). Volunteers? :-D I'm working on getting mongrel_service fixed (removing win32-service dependency since latest version broke API and affect GemPlugin namespace) I'll add the same forced ruby_version to it. I don't ask for volunteers here, but if someone can, it's really welcome :-D Regards, -- Luis Lavena Multimedia systems - A common mistake that people make when trying to design something completely foolproof is to underestimate the ingenuity of complete fools. Douglas Adams From filipe at icewall.org Thu Dec 27 07:45:24 2007 From: filipe at icewall.org (Filipe Lautert) Date: Thu, 27 Dec 2007 10:45:24 -0200 (BRST) Subject: [Mongrel-development] Review of Code for 1.9 In-Reply-To: <71166b3b0712270401q68b26a69v61c57041eac387fc@mail.gmail.com> References: <71166b3b0712251458h2227853ew2a8d652a81ce0706@mail.gmail.com> <71166b3b0712252246xf3d6503s972f83c22a008ee0@mail.gmail.com> <71166b3b0712270401q68b26a69v61c57041eac387fc@mail.gmail.com> Message-ID: On Thu, 27 Dec 2007, Luis Lavena wrote: > On Dec 27, 2007 1:07 AM, Filipe wrote: > [...] >> >> bla bla bla > > Yeah, cygwin is not a good alternative, even that some editors (like > E) encourage users to use a cygwin environment to "mimic" linux. > > I give support for mongrel on "mswin32" platform, not cygwin (hey, I > even plan to support mingw when get it working right). I installed a cygwin... gonna try to build ruby1.9 + mongrel on it today. But this thing is damn slow... say, vmware player running debian runs ruby1.9's ./configure a lot faster than cygwin. > >> Also, besides using the code already sent, we could call accept_nonblock() >> and then get into a loop calling select in the socket for a limited time and then >> check a variable to exit. Something like this: >> >> serv = TCPServer.new('127.0.0.1', 3000) >> begin >> sock = serv.accept_nonblock >> rescue Errno::EAGAIN, Errno::ECONNABORTED, Errno::EPROTO, Errno::EINTR >> #executes select for 3 seconds before try our exit variable >> IO.select([serv], nil, nil, 3) >> return if die >> retry >> end >> >> This way we just set @acceptor.die = true, join it and wait. How about it? >> > > That could work, but we will still be attached/limited/sticked/busted > to Threads. Yeah.. and it also uses select... just noticed that after I sent. Maybe the best aproach is really drop it all and rewrite this from scratch using events. Now.. which lib to use? The is event machine, zed's rev, ezmorbius' packet... lot of options. I think here wyhaines could give us a big hand! > > one developer started a thread at ruby-core listing the 'evented' > frameworks available, how bad they behave and listing part of the > problems they currently present. > > Also, describe that something could be done in the Fibers arena to > workaround this, thus removing the need for external C code: Yeap I'm following this... hope this discussion goes a long way to help us decide :) > > But still, don't have free time to look deep into this (damn work, I > hate it, need to code some usb driver for windows by tonight). :( Good luck... > > For the time being, I suggest we take stable 1.1 and: > > force required_ruby_version '<1.9.0' > release a new gems for it (I'll do the same for the pre-build binary). > > Volunteers? :-D Hey, evn is our release guy ;) > > I'm working on getting mongrel_service fixed (removing win32-service > dependency since latest version broke API and affect GemPlugin > namespace) > I'll add the same forced ruby_version to it. > > I don't ask for volunteers here, but if someone can, it's really welcome :-D Thanks for not asking... Cheers, filipe From wyhaines at gmail.com Thu Dec 27 11:47:36 2007 From: wyhaines at gmail.com (Kirk Haines) Date: Thu, 27 Dec 2007 09:47:36 -0700 Subject: [Mongrel-development] Review of Code for 1.9 In-Reply-To: References: <71166b3b0712251458h2227853ew2a8d652a81ce0706@mail.gmail.com> <71166b3b0712252246xf3d6503s972f83c22a008ee0@mail.gmail.com> <71166b3b0712270401q68b26a69v61c57041eac387fc@mail.gmail.com> Message-ID: On Dec 27, 2007 5:45 AM, Filipe Lautert wrote: > Maybe the best aproach is really drop it all and rewrite this from > scratch using events. Now.. which lib to use? The is event machine, > zed's rev, ezmorbius' packet... lot of options. > > I think here wyhaines could give us a big hand! We're in a situation where having a single version that is compatible with 1.8.x and 1.9.x means suboptimal performance on 1.9.x. So.... The choices: 1) Rewrite the network core to make use of a thread pool, and keep Mongrel, natively, as a threaded app based on core ruby facilities. 2) Rewrite the network core in an evented style of some sort. 2a) Use pure Ruby, maybe with something like hemant's packet.rb. Performance on this is not bad, but it retains the select() limitations. On the other hand, being pure ruby it'll run anywhere without any problems, including windows machines and machines that lack compilers, and will ease the packaging chores. I don't know how well packet works on jruby, but it may also ease the chore of maintaining Mongrel on that platform if it works adequately there, too. 2b) Use an event framework with an extension at its core. The main options here are really Tony Aceri's Rev, which is a brand new release, written specifically for Ruby 1.9, that uses libev under the covers, or EM. EM itself needs work before it's really 1.9 ready, though, because it's current design is optimized for best performance with 1.8 thread semantics.... Tony and I have had a couple conversations recently about the possibility of using Rev as the reactor in the pure ruby version of EM, too. The drawbacks to Rev are that is currently doesn't work on Windows, it is _very_ new, and libev, while based on libevent, is itself pretty new. I think there are a few things that are important for Mongrel. Two things, really. 1) It should run just about everywhere that Ruby does, including Windows machines. 2) It should perform pretty well everywhere that it runs. 3) It needs to remain stable. A lot of people depend on Mongrel, so we need to be conservative about changes, and do a lot of due diligence work before making any radical changes to the core. I don't think it has to run faster than anything else out there. It doesn't right now. ServerSide, for example, is a lot faster, but people host their stuff on Mongrel because Mongrel because of all of the other advantages Mongrel confers. To me, the immediate release goal should be to just repackage the current gem with a ruby version requirement on it < 1.9.0. Then, the next immediate goal should be to release a separate gem that simply works with 1.9.0. It's not a major set of changes. It doesn't matter that it doesn't work optimally so long as it does at least work. We can very, very clearly state that it's a _development_ release, and is not intended for production use. Then, third, we look at 1.9 specific changes/rewrites for performance. IMHO, we should lean towards a thread pool architecture, or a pure ruby event architecture for the base design (both of which should then work nearly unchanged on jruby, which would be good) unless developments procede in a way that lets us guarantee accessiblity of Mongrel on platforms like Windows with an event-extension based design. Kirk Haines From luislavena at gmail.com Thu Dec 27 12:30:07 2007 From: luislavena at gmail.com (Luis Lavena) Date: Thu, 27 Dec 2007 14:30:07 -0300 Subject: [Mongrel-development] Review of Code for 1.9 In-Reply-To: References: <71166b3b0712251458h2227853ew2a8d652a81ce0706@mail.gmail.com> <71166b3b0712252246xf3d6503s972f83c22a008ee0@mail.gmail.com> <71166b3b0712270401q68b26a69v61c57041eac387fc@mail.gmail.com> Message-ID: <71166b3b0712270930r5ded1ec0kbca5ff19eb777e04@mail.gmail.com> On Dec 27, 2007 1:47 PM, Kirk Haines wrote: > On Dec 27, 2007 5:45 AM, Filipe Lautert wrote: > > Maybe the best aproach is really drop it all and rewrite this from > > scratch using events. Now.. which lib to use? The is event machine, > > zed's rev, ezmorbius' packet... lot of options. > > > > I think here wyhaines could give us a big hand! > > We're in a situation where having a single version that is compatible > with 1.8.x and 1.9.x means suboptimal performance on 1.9.x. > > So.... (my comments below) > The choices: > > 1) Rewrite the network core to make use of a thread pool, and keep > Mongrel, natively, as a threaded app based on core ruby facilities. +1 > 2) Rewrite the network core in an evented style of some sort. > 2a) Use pure Ruby, maybe with something like hemant's packet.rb. > Performance on this is not bad, but it retains the select() > limitations. On the other hand, being pure ruby it'll run anywhere > without any problems, including windows machines and machines that > lack compilers, and will ease the packaging chores. I don't know how > well packet works on jruby, but it may also ease the chore of > maintaining Mongrel on that platform if it works adequately there, > too. I want to remember you that our ragel parser still requires compilation, as far we ship pre-compiled gems, we will be good on that subject. > 2b) Use an event framework with an extension at its core. The main > options here are really Tony Aceri's Rev, which is a brand new > release, written specifically for Ruby 1.9, that uses libev under the > covers, or EM. EM itself needs work before it's really 1.9 ready, > though, because it's current design is optimized for best performance > with 1.8 thread semantics.... Tony and I have had a couple > conversations recently about the possibility of using Rev as the > reactor in the pure ruby version of EM, too. The drawbacks to Rev are > that is currently doesn't work on Windows, it is _very_ new, and > libev, while based on libevent, is itself pretty new. Evented models are too radical from what Mongrel is currently now. Jumping on *any* of these frameworks (no matter if its EM or Rev) will require us a lot of effort. by pure-ruby you mean plain ruby code? It seems Rev uses a extension, which require you compile and thus, depending on it will require us (I mean: me) shipping pre-compiled versions for Windows. > I think there are a few things that are important for Mongrel. Two > things, really. > > 1) It should run just about everywhere that Ruby does, including > Windows machines. +1, we have achieved that. > 2) It should perform pretty well everywhere that it runs. It currently does, even with slowest VC6 build on Windows, Mongrel feels snappy (Mongrel, not Rails). > 3) It needs to remain stable. A lot of people depend on Mongrel, so > we need to be conservative about changes, and do a lot of due > diligence work before making any radical changes to the core. > +100 on this, I'm against huge and compulsive changes to the 'core', and pointed that several times before. > I don't think it has to run faster than anything else out there. It > doesn't right now. ServerSide, for example, is a lot faster, but > people host their stuff on Mongrel because Mongrel because of all of > the other advantages Mongrel confers. > > To me, the immediate release goal should be to just repackage the > current gem with a ruby version requirement on it < 1.9.0. > 1.1.3? > Then, the next immediate goal should be to release a separate gem that > simply works with 1.9.0. It's not a major set of changes. > It doesn't matter that it doesn't work optimally so long as it does at > least work. We can very, very clearly state that it's a _development_ > release, and is not intended for production use. > We can put these gems in mongrel.rubyforge.org/releases, to avoid users downloading them by mistake during "gem update". I need to confirm that required_ruby_version (part of rubygems specifications) is working, will do it later today. If it work as we expect, will be no need for the special /release/ and we can use rubyforge instead. > Then, third, we look at 1.9 specific changes/rewrites for performance. > IMHO, we should lean towards a thread pool architecture, or a pure > ruby event architecture for the base design (both of which should then > work nearly unchanged on jruby, which would be good) unless > developments procede in a way that lets us guarantee accessiblity of > Mongrel on platforms like Windows with an event-extension based > design. As commented before, I'm interested in Rev (and libev). I wrote my own evented library for sockets in FB almost 2 years ago, and was a great experience. Looking at C code will be really nice (nice?) after so long (without counting ruby and mongrel C code, will be 5 years). By the way Rev is structured, I don't see will be major troubles adjusting/correcting it to work on windows, but the lack of extensive test suite (or specs) and the lack of benchmarks to compare with make me think twice before invest time on this. -- Luis Lavena Multimedia systems - A common mistake that people make when trying to design something completely foolproof is to underestimate the ingenuity of complete fools. Douglas Adams From wyhaines at gmail.com Thu Dec 27 12:42:15 2007 From: wyhaines at gmail.com (Kirk Haines) Date: Thu, 27 Dec 2007 10:42:15 -0700 Subject: [Mongrel-development] Review of Code for 1.9 In-Reply-To: <71166b3b0712270930r5ded1ec0kbca5ff19eb777e04@mail.gmail.com> References: <71166b3b0712251458h2227853ew2a8d652a81ce0706@mail.gmail.com> <71166b3b0712252246xf3d6503s972f83c22a008ee0@mail.gmail.com> <71166b3b0712270401q68b26a69v61c57041eac387fc@mail.gmail.com> <71166b3b0712270930r5ded1ec0kbca5ff19eb777e04@mail.gmail.com> Message-ID: On Dec 27, 2007 10:30 AM, Luis Lavena wrote: > I want to remember you that our ragel parser still requires > compilation, as far we ship pre-compiled gems, we will be good on that > subject. Right. There is a possibility that the pure ruby one might not suck on 1.9.x, but I'm not holding my breath. However, the ragel based extension is a bit different than using an extension based event framework that itself has to work on all of the platforms that we are targetting. > by pure-ruby you mean plain ruby code? It seems Rev uses a extension, > which require you compile and thus, depending on it will require us (I > mean: me) shipping pre-compiled versions for Windows. Packet is pure ruby. That's what I was referring to. There's a pure ruby EM, but for our purposes right now, I think packet is faster (though I have not done any benchmarks on the pure ruby stuff; that's just a hunch). Kirk From luislavena at gmail.com Thu Dec 27 12:54:06 2007 From: luislavena at gmail.com (Luis Lavena) Date: Thu, 27 Dec 2007 14:54:06 -0300 Subject: [Mongrel-development] Review of Code for 1.9 In-Reply-To: References: <71166b3b0712251458h2227853ew2a8d652a81ce0706@mail.gmail.com> <71166b3b0712252246xf3d6503s972f83c22a008ee0@mail.gmail.com> <71166b3b0712270401q68b26a69v61c57041eac387fc@mail.gmail.com> <71166b3b0712270930r5ded1ec0kbca5ff19eb777e04@mail.gmail.com> Message-ID: <71166b3b0712270954u42e92499if762fb1409546010@mail.gmail.com> On Dec 27, 2007 2:42 PM, Kirk Haines wrote: > On Dec 27, 2007 10:30 AM, Luis Lavena wrote: > > > I want to remember you that our ragel parser still requires > > compilation, as far we ship pre-compiled gems, we will be good on that > > subject. > > Right. There is a possibility that the pure ruby one might not suck > on 1.9.x, but I'm not holding my breath. However, the ragel based > extension is a bit different than using an extension based event > framework that itself has to work on all of the platforms that we are > targetting. Maybe it suck less, but as I pointed in previous emails on the list, on 1.9 there wasn't "overhead" on calling C functions or pure-ruby ones. > > by pure-ruby you mean plain ruby code? It seems Rev uses a extension, > > which require you compile and thus, depending on it will require us (I > > mean: me) shipping pre-compiled versions for Windows. > > Packet is pure ruby. That's what I was referring to. There's a pure > ruby EM, but for our purposes right now, I think packet is faster > (though I have not done any benchmarks on the pure ruby stuff; that's > just a hunch). Based on rumors (comments on ruby-core) it seems EM is not designed for 1.9 and do a intense use of rb_thread_schedule and rb_funcall -- maybe I should call it abuse ;-) Damn, I really hate we can do so many things to test and research but not enough time to actually do it! :-P -- Luis Lavena Multimedia systems - A common mistake that people make when trying to design something completely foolproof is to underestimate the ingenuity of complete fools. Douglas Adams From wyhaines at gmail.com Thu Dec 27 13:09:54 2007 From: wyhaines at gmail.com (Kirk Haines) Date: Thu, 27 Dec 2007 11:09:54 -0700 Subject: [Mongrel-development] Review of Code for 1.9 In-Reply-To: <71166b3b0712270954u42e92499if762fb1409546010@mail.gmail.com> References: <71166b3b0712251458h2227853ew2a8d652a81ce0706@mail.gmail.com> <71166b3b0712252246xf3d6503s972f83c22a008ee0@mail.gmail.com> <71166b3b0712270401q68b26a69v61c57041eac387fc@mail.gmail.com> <71166b3b0712270930r5ded1ec0kbca5ff19eb777e04@mail.gmail.com> <71166b3b0712270954u42e92499if762fb1409546010@mail.gmail.com> Message-ID: > Based on rumors (comments on ruby-core) it seems EM is not designed > for 1.9 and do a intense use of rb_thread_schedule and rb_funcall -- > maybe I should call it abuse ;-) Regarding 1.9, I mentioned that in my earlier email. It was designed to work optimally with the 1.8 threading model. What this involves is making calls to rb_thread_select at specific points in the event loop so that Ruby has a chance to do its thing. rb_funcall comes into play when events are being triggered. The architecture isn't really an abuse of rb_funcall. What happens is that there is an event_callback method that the extension calls. The event_callback method is in pure ruby, and it, in turn, dispatches the event callbacks that one wrote in one's protocol class. If Ruby's method dispatch were faster, it'd be a non-issue, and would constitute a nice, modular design. Because Ruby's method dispatch isn't fast, though, I have an optimization that I am waiting to commit that more than halves the amount of time taken to dispatch events. > Damn, I really hate we can do so many things to test and research but > not enough time to actually do it! :-P Yeah. I know. :( A person could _easily_ work full time on this sort of stuff, if there were money in it. Kirk From filipe at icewall.org Thu Dec 27 13:22:53 2007 From: filipe at icewall.org (Filipe Lautert) Date: Thu, 27 Dec 2007 16:22:53 -0200 (BRST) Subject: [Mongrel-development] Review of Code for 1.9 In-Reply-To: References: <71166b3b0712251458h2227853ew2a8d652a81ce0706@mail.gmail.com> <71166b3b0712252246xf3d6503s972f83c22a008ee0@mail.gmail.com> <71166b3b0712270401q68b26a69v61c57041eac387fc@mail.gmail.com> <71166b3b0712270930r5ded1ec0kbca5ff19eb777e04@mail.gmail.com> <71166b3b0712270954u42e92499if762fb1409546010@mail.gmail.com> Message-ID: On Thu, 27 Dec 2007, Kirk Haines wrote: > ... >> Damn, I really hate we can do so many things to test and research but >> not enough time to actually do it! :-P > > Yeah. I know. :( A person could _easily_ work full time on this > sort of stuff, if there were money in it. Yeap, sad. Someone from the team could have vacations to do it ;) Anyway, I really liked the idea of release 1.1.3 for ruby < 1.9.0, release a version that just works on ruby 1.9 and play new core things to a new major release (2.0?) . Also, we have some things that could be released in a 1.2 release, like logging facility. So, by the talks that I see for 1.9 support here in the list, it seems that we are really going to an event based/thread based approach, is it? And the preference is for a pure ruby approach. Got it? Cheers, Filipe From evan at cloudbur.st Thu Dec 27 15:14:57 2007 From: evan at cloudbur.st (Evan Weaver) Date: Thu, 27 Dec 2007 15:14:57 -0500 Subject: [Mongrel-development] Review of Code for 1.9 In-Reply-To: References: <71166b3b0712251458h2227853ew2a8d652a81ce0706@mail.gmail.com> <71166b3b0712270401q68b26a69v61c57041eac387fc@mail.gmail.com> <71166b3b0712270930r5ded1ec0kbca5ff19eb777e04@mail.gmail.com> <71166b3b0712270954u42e92499if762fb1409546010@mail.gmail.com> Message-ID: I'm gonna see if CNET can give me some official time for Mongrel work. Dunno if it will happen though. Evan On Dec 27, 2007 1:22 PM, Filipe Lautert wrote: > On Thu, 27 Dec 2007, Kirk Haines wrote: > > ... > >> Damn, I really hate we can do so many things to test and research but > >> not enough time to actually do it! :-P > > > > Yeah. I know. :( A person could _easily_ work full time on this > > sort of stuff, if there were money in it. > > Yeap, sad. Someone from the team could have vacations to do it ;) > > Anyway, I really liked the idea of release 1.1.3 for ruby < 1.9.0, > release a version that just works on ruby 1.9 and play new core things to > a new major release (2.0?) . > > Also, we have some things that could be released in a 1.2 release, like > logging facility. > > So, by the talks that I see for 1.9 support here in the list, it seems that > we are really going to an event based/thread based approach, is it? And > the preference is for a pure ruby approach. Got it? > > Cheers, > > Filipe > > _______________________________________________ > Mongrel-development mailing list > Mongrel-development at rubyforge.org > http://rubyforge.org/mailman/listinfo/mongrel-development > -- Evan Weaver Cloudburst, LLC From wayneeseguin at gmail.com Thu Dec 27 20:47:05 2007 From: wayneeseguin at gmail.com (Wayne E. Seguin) Date: Thu, 27 Dec 2007 20:47:05 -0500 Subject: [Mongrel-development] Review of Code for 1.9 In-Reply-To: References: <71166b3b0712251458h2227853ew2a8d652a81ce0706@mail.gmail.com> <71166b3b0712270401q68b26a69v61c57041eac387fc@mail.gmail.com> <71166b3b0712270930r5ded1ec0kbca5ff19eb777e04@mail.gmail.com> <71166b3b0712270954u42e92499if762fb1409546010@mail.gmail.com> Message-ID: I am in favor of: * Keeping as much of Mongrel core as it is, if possible. * Rewriting only what is needed to target a thread pool (configurable) and/or evented implementation (I believe that adding support for both would be ideal). ~Wayne -------------- next part -------------- An HTML attachment was scrubbed... URL: http://rubyforge.org/pipermail/mongrel-development/attachments/20071227/e1ee52f8/attachment.html From zedshaw at zedshaw.com Thu Dec 27 21:57:57 2007 From: zedshaw at zedshaw.com (Zed A. Shaw) Date: Thu, 27 Dec 2007 21:57:57 -0500 Subject: [Mongrel-development] Review of Code for 1.9 In-Reply-To: <71166b3b0712270930r5ded1ec0kbca5ff19eb777e04@mail.gmail.com> References: <71166b3b0712251458h2227853ew2a8d652a81ce0706@mail.gmail.com> <71166b3b0712252246xf3d6503s972f83c22a008ee0@mail.gmail.com> <71166b3b0712270401q68b26a69v61c57041eac387fc@mail.gmail.com> <71166b3b0712270930r5ded1ec0kbca5ff19eb777e04@mail.gmail.com> Message-ID: <20071227215757.2530d2f7.zedshaw@zedshaw.com> On Thu, 27 Dec 2007 14:30:07 -0300 "Luis Lavena" wrote: > On Dec 27, 2007 1:47 PM, Kirk Haines wrote: > > On Dec 27, 2007 5:45 AM, Filipe Lautert wrote: > > > Maybe the best aproach is really drop it all and rewrite this from > > > scratch using events. Now.. which lib to use? The is event machine, > > > zed's rev, ezmorbius' packet... lot of options. No, that's a bad idea, and again I keep saying and again people keep suggesting it so I'll say it one more time: Use only what's available in Ruby from the beginning (select and threads) for the base mongrel IO processing, and then have a mod for other methods. There's a reason: Other event systems and libs that are not part of ruby will get stomped on by all the different Ruby implementations, and if a new ruby comes out that isn't retarded you'll all get bit in the ass trying to use it. That means, use regular ruby with minimal gear for the base mongrel. Not evented libraries. Additionally: Event based systems are not a cure for speed. They are a lame hack for getting around most operating system's shitty AIO or thread implementations. If you don't always have your vanilla source tree you'll get screwed when these systems improve. It happens all the time so learn from past mistakes. And finally: Mongrel is also a library. The problem with event libs is once you use them they are viral and always require everything else to start using the event system. If the only way someone can embed mongrel in their little app is to also include all of EM or Rev and then hook their crap into that event loop then you've all failed hard. > > We're in a situation where having a single version that is compatible > > with 1.8.x and 1.9.x means suboptimal performance on 1.9.x. > > > > So.... Sigh, metrics and evidence please. Without those a debate on performance is meaningless. > > The choices: > > > > 1) Rewrite the network core to make use of a thread pool, and keep > > Mongrel, natively, as a threaded app based on core ruby facilities. Now, this is doable, and I did this intially, so I'll lay out some original design choices I tried and why I didn't do them: 1) select based reactor to a state machine based processor. Worked great and was fast, but crashed HARD when you put it in a thread since ruby confused using select in a thread with having to deadlocked threads. This is because ruby (used to?) not properly mesh the thread select processing with regular select calls so juggling this was a nightmare. This was actually the absolute fastest thing I built except for this last flaw. 2) Thread pool reading of a single work queue at random with a feeder thread. This was not fast at all. The thread queues were so slow then and buggy that they were next to useless, and they aslo didn't allocate to the readers at random. Also the contention between the main master read/accept thread and the other threads made this not work so well. 3) Select loop to read the request, then put that on a thread queue. Combined both problems for a spectacular disaster. 4) Create new thread for each request. This is what you have now. So, I'd say try these again, with the select+FSM approach first and then see what you get. The #1 approach worked great as long as you didn't put the select inside a thread, and it'd basically be a lo-fi event system that everyone can love. A thread queue would be next but you'd probably have a lot of work to do on that. > > I think there are a few things that are important for Mongrel. Two > > things, really. > > > > 1) It should run just about everywhere that Ruby does, including > > Windows machines. > > +1, we have achieved that. Not really, the requirement is that it run embedded inside anything with just a few little calls and a thread. See above for why EM, Rev, or Packrat don't satisfy that. > > To me, the immediate release goal should be to just repackage the > > current gem with a ruby version requirement on it < 1.9.0. > > > > 1.1.3? That's what I'd shoot for in the next release, also, what happened to giving people the ablity to put in their own "work handler" so that this problem becomes somebody else's problem? -- Zed A. Shaw - Hate: http://savingtheinternetwithhate.com/ - Good: http://www.zedshaw.com/ - Evil: http://yearofevil.com/ From filipe at icewall.org Fri Dec 28 00:10:14 2007 From: filipe at icewall.org (Filipe Lautert) Date: Fri, 28 Dec 2007 03:10:14 -0200 (BRST) Subject: [Mongrel-development] Review of Code for 1.9 In-Reply-To: References: <71166b3b0712251458h2227853ew2a8d652a81ce0706@mail.gmail.com> <71166b3b0712270401q68b26a69v61c57041eac387fc@mail.gmail.com> <71166b3b0712270930r5ded1ec0kbca5ff19eb777e04@mail.gmail.com> <71166b3b0712270954u42e92499if762fb1409546010@mail.gmail.com> Message-ID: On Thu, 27 Dec 2007, Wayne E. Seguin wrote: > I am in favor of: > > * Keeping as much of Mongrel core as it is, if possible. > * Rewriting only what is needed to target a thread pool (configurable) > and/or evented implementation (I believe that adding support for both would > be ideal). WHOW, now here is a new thing for me. wyhaines and zed agreeing about something! Then let us play with threads! Btw, with commits from today to trunk, just one test is not running on ruby 1.9 . But I'm already discuting this at ruby-core list (I wrote a resume at my blog anyway), because this is a Net::HTTP problem, and not a mongrel one. So, by now trunk builds in 1.8 and 1.9 and passes all tests (except this one) for both! Cheers, filipe { @ icewall.org GPG 1024D/A6BA423E http://filipe.icewall.org/ } From luislavena at gmail.com Fri Dec 28 08:59:52 2007 From: luislavena at gmail.com (Luis Lavena) Date: Fri, 28 Dec 2007 10:59:52 -0300 Subject: [Mongrel-development] Review of Code for 1.9 In-Reply-To: <20071227215757.2530d2f7.zedshaw@zedshaw.com> References: <71166b3b0712251458h2227853ew2a8d652a81ce0706@mail.gmail.com> <71166b3b0712252246xf3d6503s972f83c22a008ee0@mail.gmail.com> <71166b3b0712270401q68b26a69v61c57041eac387fc@mail.gmail.com> <71166b3b0712270930r5ded1ec0kbca5ff19eb777e04@mail.gmail.com> <20071227215757.2530d2f7.zedshaw@zedshaw.com> Message-ID: <71166b3b0712280559k1403fc23s7161c8c2354b2b7f@mail.gmail.com> On Dec 27, 2007 11:57 PM, Zed A. Shaw wrote: > > > > We're in a situation where having a single version that is compatible > > > with 1.8.x and 1.9.x means suboptimal performance on 1.9.x. > > > > > > So.... > > Sigh, metrics and evidence please. Without those a debate on > performance is meaningless. > I can confirm that thread spawning and context switching in 1.9 is more expensive than 1.8, done some tests but didn't collect the data for posterity (or for later big discussion and metrics comparison with you Zed) :-D Also, done some testing regarding C function calling and pure-ruby function calling overhead, being 1.9 slower than 1.8 in pure-ruby calling. > > > The choices: > > > > > > 1) Rewrite the network core to make use of a thread pool, and keep > > > Mongrel, natively, as a threaded app based on core ruby facilities. > > Now, this is doable, and I did this intially, so I'll lay out some > original design choices I tried and why I didn't do them: > > 1) select based reactor to a state machine based processor. Worked > great and was fast, but crashed HARD when you put it in a thread since > ruby confused using select in a thread with having to deadlocked > threads. This is because ruby (used to?) not properly mesh the thread > select processing with regular select calls so juggling this was a > nightmare. This was actually the absolute fastest thing I built > except for this last flaw. > Did you keep that original code somewhere? A few things improved regarding this topic, or at least they seem in 1.9 > 2) Thread pool reading of a single work queue at random with a feeder > thread. This was not fast at all. The thread queues were so slow > then and buggy that they were next to useless, and they aslo didn't > allocate to the readers at random. Also the contention between the > main master read/accept thread and the other threads made this not work > so well. > Can't comment on this, my brain is still trying to understand what you wrote. > 3) Select loop to read the request, then put that on a thread queue. > Combined both problems for a spectacular disaster. > > 4) Create new thread for each request. This is what you have now. > This is resource expensive, I'll collect some sample data this weekend and show to you the problem that is present in 1.9 regarding this. I've followed that path before, but on FreeBASIC, and faced the same results, FB threads where native (linux and windows) but wasn't scalable, too many threads bring the OS down, linux or windows. Didn't test it on Solaris, which have a better threading functionality to compare with. > > > I think there are a few things that are important for Mongrel. Two > > > things, really. > > > > > > 1) It should run just about everywhere that Ruby does, including > > > Windows machines. > > > > +1, we have achieved that. > > Not really, the requirement is that it run embedded inside anything > with just a few little calls and a thread. See above for why EM, Rev, > or Packrat don't satisfy that. > I was talking about *current* mongrel, not the hypothetical evented one. > > > To me, the immediate release goal should be to just repackage the > > > current gem with a ruby version requirement on it < 1.9.0. > > > > > > > 1.1.3? > > That's what I'd shoot for in the next release, also, what happened to > giving people the ablity to put in their own "work handler" so that > this problem becomes somebody else's problem? Mmm, don't follow you on that. You mean allow pluggable replacement for HttpServer? -- Luis Lavena Multimedia systems - A common mistake that people make when trying to design something completely foolproof is to underestimate the ingenuity of complete fools. Douglas Adams From wyhaines at gmail.com Fri Dec 28 10:02:20 2007 From: wyhaines at gmail.com (Kirk Haines) Date: Fri, 28 Dec 2007 08:02:20 -0700 Subject: [Mongrel-development] Review of Code for 1.9 In-Reply-To: <71166b3b0712280559k1403fc23s7161c8c2354b2b7f@mail.gmail.com> References: <71166b3b0712251458h2227853ew2a8d652a81ce0706@mail.gmail.com> <71166b3b0712252246xf3d6503s972f83c22a008ee0@mail.gmail.com> <71166b3b0712270401q68b26a69v61c57041eac387fc@mail.gmail.com> <71166b3b0712270930r5ded1ec0kbca5ff19eb777e04@mail.gmail.com> <20071227215757.2530d2f7.zedshaw@zedshaw.com> <71166b3b0712280559k1403fc23s7161c8c2354b2b7f@mail.gmail.com> Message-ID: On Dec 28, 2007 6:59 AM, Luis Lavena wrote: > > That's what I'd shoot for in the next release, also, what happened to > > giving people the ablity to put in their own "work handler" so that > > this problem becomes somebody else's problem? > > Mmm, don't follow you on that. You mean allow pluggable replacement > for HttpServer? I think Zed is referring to the suggestion that the portion of the guts that deals with network IO interactions, and the idea that we make that more modular so that it is easy to plug different architectures into it. Kirk Haines From evan at cloudbur.st Fri Dec 28 19:23:27 2007 From: evan at cloudbur.st (Evan Weaver) Date: Fri, 28 Dec 2007 19:23:27 -0500 Subject: [Mongrel-development] Review of Code for 1.9 In-Reply-To: References: <71166b3b0712251458h2227853ew2a8d652a81ce0706@mail.gmail.com> <71166b3b0712252246xf3d6503s972f83c22a008ee0@mail.gmail.com> <71166b3b0712270401q68b26a69v61c57041eac387fc@mail.gmail.com> <71166b3b0712270930r5ded1ec0kbca5ff19eb777e04@mail.gmail.com> <20071227215757.2530d2f7.zedshaw@zedshaw.com> <71166b3b0712280559k1403fc23s7161c8c2354b2b7f@mail.gmail.com> Message-ID: Zed, Can you elaborate on the thread pool problems you had? Do you think any of these will be mitigated by YARV threads (better scheduling etc). Evan On Dec 28, 2007 10:02 AM, Kirk Haines wrote: > On Dec 28, 2007 6:59 AM, Luis Lavena wrote: > > > > That's what I'd shoot for in the next release, also, what happened to > > > giving people the ablity to put in their own "work handler" so that > > > this problem becomes somebody else's problem? > > > > Mmm, don't follow you on that. You mean allow pluggable replacement > > for HttpServer? > > I think Zed is referring to the suggestion that the portion of the > guts that deals with network IO interactions, and the idea that we > make that more modular so that it is easy to plug different > architectures into it. > > > Kirk Haines > > _______________________________________________ > Mongrel-development mailing list > Mongrel-development at rubyforge.org > http://rubyforge.org/mailman/listinfo/mongrel-development > -- Evan Weaver Cloudburst, LLC From zedshaw at zedshaw.com Fri Dec 28 21:12:14 2007 From: zedshaw at zedshaw.com (Zed A. Shaw) Date: Fri, 28 Dec 2007 21:12:14 -0500 Subject: [Mongrel-development] Review of Code for 1.9 In-Reply-To: References: <71166b3b0712251458h2227853ew2a8d652a81ce0706@mail.gmail.com> <71166b3b0712252246xf3d6503s972f83c22a008ee0@mail.gmail.com> <71166b3b0712270401q68b26a69v61c57041eac387fc@mail.gmail.com> <71166b3b0712270930r5ded1ec0kbca5ff19eb777e04@mail.gmail.com> <20071227215757.2530d2f7.zedshaw@zedshaw.com> <71166b3b0712280559k1403fc23s7161c8c2354b2b7f@mail.gmail.com> Message-ID: <20071228211214.1856a5a4.zedshaw@zedshaw.com> On Fri, 28 Dec 2007 19:23:27 -0500 "Evan Weaver" wrote > Zed, > > Can you elaborate on the thread pool problems you had? Do you think > any of these will be mitigated by YARV threads (better scheduling > etc). They might be gone in YARV since the threading is more real and (supposedly) higher performance. My problem was mostly that putting the requests through a thread queue was a lot slower (1/2 speed iirc) than just making new threads. Without a queue though it's really difficult to do a pool of processors model. I tried using the plain synchronized primitives but that turned out to be the same problem for speed, especially when combined with rails. Finally, in order for a set of queue listeners to work properly the selection of the next processor/listener to receive a queue entry has to be fairly random. However I found that it wasn't that random and usually it was more first-come-first served. -- Zed A. Shaw - Hate: http://savingtheinternetwithhate.com/ - Good: http://www.zedshaw.com/ - Evil: http://yearofevil.com/ From zedshaw at zedshaw.com Fri Dec 28 21:31:08 2007 From: zedshaw at zedshaw.com (Zed A. Shaw) Date: Fri, 28 Dec 2007 21:31:08 -0500 Subject: [Mongrel-development] [SECURITY] Patch For Bug Serving Arbitrary Files Message-ID: <20071228213108.35b0c427.zedshaw@zedshaw.com> This is a proposed patch for the security hole reported today. You can just add the test for @path being at index 0 in the exanded req_path as shown below. Take heed of the comment I've added too, and there was a test for this very attack in the unit test suite, so it was removed by someone as well. I didn't test this but I'm pretty sure it's the fix. === lib/mongrel/handlers.rb ================================================================== --- lib/mongrel/handlers.rb (revision 6851) +++ lib/mongrel/handlers.rb (local) @@ -132,8 +132,12 @@ # Add the drive letter or root path req_path = File.join(@path, req_path) if @path req_path = File.expand_path req_path - - if File.exist? req_path + + # do not remove the check for @path at the beginning, it's what prevents + # the serving of arbitrary files (and good programmer Rule #1 Says: If + # you don't understand something, it's not because I'm stupid, it's + # because you are). + if req_path.index(@path) == 0 and File.exist? req_path # It exists and it's in the right location if File.directory? req_path # The request is for a directory @@ -153,7 +157,7 @@ return req_path end else - # does not exist or isn't in the right spot + # does not exist or isn't in the right spot or isn't valid because not start with @path return nil end end -- Zed A. Shaw - Hate: http://savingtheinternetwithhate.com/ - Good: http://www.zedshaw.com/ - Evil: http://yearofevil.com/