From normalperson at yhbt.net Wed Aug 12 21:02:23 2009 From: normalperson at yhbt.net (Eric Wong) Date: Wed, 12 Aug 2009 18:02:23 -0700 Subject: [Rev-talk] making listeners from existing objects (w/preliminary patch) Message-ID: <20090813010222.GA22881@dcvr.yhbt.net> Hi, I'm toying around with projects that receive/inherit file descriptors from other processes. So in those cases I'll end up with a ::TCPServer object already in my hands (usually from ::TCPServer.for_fd(fileno)) that I'd like to use with Revactor. The following is a first step/patch into making rev/revactor able to accept existing server objects. See examples t0.rb + t1.rb below on how they'd be used). diff --git a/lib/rev/listener.rb b/lib/rev/listener.rb index 9b17ce7..aedc11a 100644 --- a/lib/rev/listener.rb +++ b/lib/rev/listener.rb @@ -49,11 +49,11 @@ module Rev # :backlog - Max size of the pending connection queue (default 1024) # :reverse_lookup - Retain BasicSocket's reverse DNS functionality (default false) # - def initialize(addr, port, options = {}) + def initialize(addr, port = nil, options = {}) BasicSocket.do_not_reverse_lookup = true unless options[:reverse_lookup] options[:backlog] ||= DEFAULT_BACKLOG - listen_socket = ::TCPServer.new(addr, port) + listen_socket = ::TCPServer === addr ? addr : ::TCPServer.new(addr, port) listen_socket.instance_eval { listen(options[:backlog]) } super(listen_socket) end --- In this example, t0.rb binds the socket (127.0.0.1:8888), and then executes t1.rb immediately. t1.rb does the actual work with the socket that t0.rb bound. ------------------ t0.rb ------------------- #!/home/ew/ruby-1.9/bin/ruby require 'socket' s = TCPServer.new('127.0.0.1', 8888) # give the child process a file descriptor to inherit: exec './t1.rb', s.fileno.to_s ------------------ t1.rb ------------------- #!/home/ew/ruby-1.9/bin/ruby require 'revactor' fileno = ARGV.first.to_i s = TCPServer.for_fd(fileno) listener = Revactor::TCP.listen(s, nil) p [ :inherited, listener ] # the echo server example shipped with revactor: loop do # Accept an incoming connection and start a new Actor # to handle it Actor.spawn(listener.accept) do |sock| puts "#{sock.remote_addr}:#{sock.remote_port} connected" # Begin echoing received data loop do begin # Write everything we read sock.write sock.read rescue EOFError puts "#{sock.remote_addr}:#{sock.remote_port} disconnected" # Break (and exit the current actor) if the connection # is closed, just like with a normal Ruby socket break end end end end --- Please let me know what you think, thanks! I'll make a proper patch+commit message that also works for UNIXServer if you all like it. -- Eric Wong From normalperson at yhbt.net Thu Aug 13 03:21:34 2009 From: normalperson at yhbt.net (Eric Wong) Date: Thu, 13 Aug 2009 00:21:34 -0700 Subject: [Rev-talk] making listeners from existing objects (w/preliminary patch) In-Reply-To: References: <20090813010222.GA22881@dcvr.yhbt.net> Message-ID: <20090813072134.GA32240@dcvr.yhbt.net> Tony Arcieri wrote: > That looks mostly good, aside from the fact that there's no check for if a > user passes a bind address but no port. If you'd like to put together a > patch or send me a pull request on Github I can incorporate this. I've added the check and also made UNIXListener behave the same way I've also added a spec for UNIXListener since it's easier to test UNIXListener than TCPListener. I've pushed out to git://yhbt.net/rev.git. I'm not comfortable using Github for multiple reasons. I'll also reply to this message with the patches so you can comment on them if needed. There's also a second patch in the series that I think will be useful: Eric Wong (2): Allow listeners to build off existing {UNIX,TCP}Server objects Add Rev::Listener#fileno to access the numeric descriptor --- lib/rev/listener.rb | 23 +++++++++++++++++++---- spec/unix_listener_spec.rb | 25 +++++++++++++++++++++++++ 2 files changed, 44 insertions(+), 4 deletions(-) -- Eric Wong From normalperson at yhbt.net Thu Aug 13 03:22:22 2009 From: normalperson at yhbt.net (Eric Wong) Date: Thu, 13 Aug 2009 00:22:22 -0700 Subject: [Rev-talk] [PATCH 1/2] Allow listeners to build off existing {UNIX, TCP}Server objects In-Reply-To: <20090813072134.GA32240@dcvr.yhbt.net> References: <20090813010222.GA22881@dcvr.yhbt.net> <20090813072134.GA32240@dcvr.yhbt.net> Message-ID: <20090813072222.GA32614@dcvr.yhbt.net> Instead of having these objects to bind new descriptors at initialization, allow them to be initialized off existing TCPServer or UNIXServer objects. This is useful for servers that are handed numeric file descriptors and create *Server objects out of them via {TCP,UNIX}Server.for_fd Signed-off-by: Eric Wong --- lib/rev/listener.rb | 16 +++++++++++++--- spec/unix_listener_spec.rb | 27 +++++++++++++++++++++++++++ 2 files changed, 40 insertions(+), 3 deletions(-) create mode 100644 spec/unix_listener_spec.rb diff --git a/lib/rev/listener.rb b/lib/rev/listener.rb index 9b17ce7..3634d95 100644 --- a/lib/rev/listener.rb +++ b/lib/rev/listener.rb @@ -49,11 +49,19 @@ module Rev # :backlog - Max size of the pending connection queue (default 1024) # :reverse_lookup - Retain BasicSocket's reverse DNS functionality (default false) # - def initialize(addr, port, options = {}) + # If the specified address is an TCPServer object, it will ignore + # the port and :backlog option and create a new Rev::TCPListener out + # of the existing TCPServer object. + def initialize(addr, port = nil, options = {}) BasicSocket.do_not_reverse_lookup = true unless options[:reverse_lookup] options[:backlog] ||= DEFAULT_BACKLOG - listen_socket = ::TCPServer.new(addr, port) + listen_socket = if ::TCPServer === addr + addr + else + raise ArgumentError, "port must be an integer" if nil == port + ::TCPServer.new(addr, port) + end listen_socket.instance_eval { listen(options[:backlog]) } super(listen_socket) end @@ -63,8 +71,10 @@ module Rev # Create a new Rev::UNIXListener # # Accepts the same arguments as UNIXServer.new + # Optionally, it can also take anyn existing UNIXServer object + # and create a Rev::UNIXListener out of it. def initialize(*args) - super(::UNIXServer.new(*args)) + super(::UNIXServer === args.first ? args.first : ::UNIXServer.new(*args)) end end end diff --git a/spec/unix_listener_spec.rb b/spec/unix_listener_spec.rb new file mode 100644 index 0000000..281d149 --- /dev/null +++ b/spec/unix_listener_spec.rb @@ -0,0 +1,27 @@ +require File.dirname(__FILE__) + '/../lib/rev' +require 'tempfile' + +describe Rev::UNIXListener do + + before :each do + @tmp = Tempfile.new('rev_unix_listener_spec') + File.unlink(@tmp.path).should == 1 + File.exist?(@tmp.path).should == false + end + + it "creates a new UNIXListener" do + listener = Rev::UNIXListener.new(@tmp.path) + File.socket?(@tmp.path).should == true + end + + it "builds off an existing UNIXServer" do + unix_server = UNIXServer.new(@tmp.path) + File.socket?(@tmp.path).should == true + listener = Rev::UNIXListener.new(unix_server) + File.socket?(@tmp.path).should == true + listener.instance_eval { + @listen_socket.fileno.should == unix_server.fileno + } + end + +end -- Eric Wong From normalperson at yhbt.net Thu Aug 13 03:22:51 2009 From: normalperson at yhbt.net (Eric Wong) Date: Thu, 13 Aug 2009 00:22:51 -0700 Subject: [Rev-talk] [PATCH 2/2] Add Rev::Listener#fileno to access the numeric descriptor In-Reply-To: <20090813072134.GA32240@dcvr.yhbt.net> References: <20090813010222.GA22881@dcvr.yhbt.net> <20090813072134.GA32240@dcvr.yhbt.net> Message-ID: <20090813072251.GB32614@dcvr.yhbt.net> It is useful to access the underlying file descriptor in case a server needs to exec() a new process and share the file descriptor information with its children (via the command-line or environment variable). The child could then call *Server.for_fd on the file descriptor. This also makes the existing unix_listener_spec easier to implement. Signed-off-by: Eric Wong --- lib/rev/listener.rb | 7 ++++++- spec/unix_listener_spec.rb | 4 +--- 2 files changed, 7 insertions(+), 4 deletions(-) diff --git a/lib/rev/listener.rb b/lib/rev/listener.rb index 3634d95..7069737 100644 --- a/lib/rev/listener.rb +++ b/lib/rev/listener.rb @@ -15,7 +15,12 @@ module Rev @listen_socket = listen_socket super(@listen_socket) end - + + # Returns an integer representing the underlying numeric file descriptor + def fileno + @listen_socket.fileno + end + # Close the listener def close detach if attached? diff --git a/spec/unix_listener_spec.rb b/spec/unix_listener_spec.rb index 281d149..a264cf7 100644 --- a/spec/unix_listener_spec.rb +++ b/spec/unix_listener_spec.rb @@ -19,9 +19,7 @@ describe Rev::UNIXListener do File.socket?(@tmp.path).should == true listener = Rev::UNIXListener.new(unix_server) File.socket?(@tmp.path).should == true - listener.instance_eval { - @listen_socket.fileno.should == unix_server.fileno - } + listener.fileno.should == unix_server.fileno end end -- Eric Wong From normalperson at yhbt.net Sun Aug 23 02:10:37 2009 From: normalperson at yhbt.net (Eric Wong) Date: Sat, 22 Aug 2009 23:10:37 -0700 Subject: [Rev-talk] [PATCH 2/2] Add Rev::Listener#fileno to access the numeric descriptor In-Reply-To: References: <20090813010222.GA22881@dcvr.yhbt.net> <20090813072134.GA32240@dcvr.yhbt.net> <20090813072251.GB32614@dcvr.yhbt.net> Message-ID: <20090823061036.GA7289@dcvr.yhbt.net> Tony Arcieri wrote: > Hi, sorry about the belated reply. > > I can't get this patch to reply. Can you fork my repository on github and > send me a pull request, or attach the patchfile as an attachment rather than > pasting it into the body of the email? Yeah, in the other email I noted I also cloned and pushed to git://yhbt.net/rev.git I'll continue pushing to that if I come up with other patches. > patching file lib/rev/listener.rb > Hunk #1 FAILED at 15. > 1 out of 1 hunk FAILED -- saving rejects to file lib/rev/listener.rb.rej > patching file spec/unix_listener_spec.rb > Hunk #1 FAILED at 19. > 1 out of 1 hunk FAILED -- saving rejects to file > spec/unix_listener_spec.rb.rej Odd, which email client do you use? Some folks I know save the emails to a temporary mbox and run "git am" against it. I've also been using mutt with the following macros defined in my muttrc for the past few years with great success: macro index A ":unset pipe_decode\n|git am -3\n:set pipe_decode\n" macro pager A ":unset pipe_decode\n|git am -3\n:set pipe_decode\n" Basically the raw email message gets piped to "git am" (instead of patch(1)) so the entire commit message + authorship information goes into git (I run mutt inside my source trees). I never have to leave my email client to apply a patch :) -- Eric Wong From normalperson at yhbt.net Thu Aug 27 16:12:02 2009 From: normalperson at yhbt.net (Eric Wong) Date: Thu, 27 Aug 2009 13:12:02 -0700 Subject: [Rev-talk] Prepping for rev 0.3.0 In-Reply-To: References: Message-ID: <20090827201202.GB3185@dcvr.yhbt.net> Tony Arcieri wrote: > I am preparing to release rev 0.3.0 with the following changes: > - Added Rev::Listener#fileno for accessing the underlying file descriptor > > - Support for creating Rev::Listeners from existing TCPServers/UNIXServers Cool. I've just pushed out similar changes to Rev::Server that I may need in the near future: Eric Wong (2): Rev::UNIXServer: use path instead of the first argument Rev::Server-based classes can build off ::*Server objects lib/rev/server.rb | 21 ++++++++++++++++++--- spec/unix_server_spec.rb | 25 +++++++++++++++++++++++++ 2 files changed, 43 insertions(+), 3 deletions(-) Pullable from git://yhbt.net/rev.git (browsable from http://git.bogomips.org/cgit/rev.git, too) > - Upgrade to libev 3.8 > - Uses iobuffer gem instead of its own Rev::Buffer class. This gem is now > optimized so it no longer makes any system calls aside from read/write, > whereas the previous one needlessly polled for the current time in many > cases (as part of its memory pooling). I just tried installing iobuffer under 1.8.7 and got this: cc -I. -I. -I/home/ew/lib/ruby/1.8/x86_64-linux -I. -fPIC -pipe -Wall -O2 -momit-leaf-frame-pointer -c iobuffer.c iobuffer.c: In function ?IO_Buffer_read_from?: iobuffer.c:292: error: ?rb_io_t? has no member named ?fd? iobuffer.c: In function ?IO_Buffer_write_to?: iobuffer.c:315: error: ?rb_io_t? has no member named ?fd? make: *** [iobuffer.o] Error 1 It installs fine under 1.8.6 and 1.9.1. Unfortunately many people got switched to 1.8.7 by their vendor... Not sure if you care. > As you may or may not know, Rev's test suite is virtually nonexistent :( :( > :(. If you are using Rev, I'd love for you to test out HEAD off github and > let me know if it works for you: I need to get around to working on projects that use Rev :) Those will eventually have fairly complete (though probably strange) integration tests. I'm not a huge fan of unit tests (especially not of mocking), for network servers/clients. I'd rather just fire up a whole server/client combo. Taking TCP ports and expecting them to run right all the time can be a pain as well as well. I've been using the following bit for some projects I've been working on: ----------------------------- 8< -------------------------- # unused_port provides an unused port on +addr+ usable for TCP that is # guaranteed to be unused across all xxxxxx builds on that system. It # prevents race conditions by using a lock file other xxxxxx builds # will see. This is required if you perform several builds in parallel # with a continuous integration system or run tests in parallel via # gmake. This is NOT guaranteed to be race-free if you run other # processes that bind to random ports for testing (but the window # for a race condition is very small). You may also set XXXXXX_TEST_ADDR # to override the default test address (127.0.0.1). def unused_port(addr = '127.0.0.1') retries = 100 base = 5000 port = sock = nil begin begin port = base + rand(32768 - base) sock = Socket.new(Socket::AF_INET, Socket::SOCK_STREAM, 0) sock.bind(Socket.pack_sockaddr_in(port, addr)) sock.listen(5) rescue Errno::EADDRINUSE, Errno::EACCES sock.close rescue nil retry if (retries -= 1) >= 0 end # since we'll end up closing the random port we just got, there's a race # condition could allow the random port we just chose to reselect itself # when running tests in parallel with gmake. Create a lock file while # we have the port here to ensure that does not happen . lock_path = "#{Dir::tmpdir}/xxxxxx_test.#{addr}:#{port}.lock" lock = File.open(lock_path, File::WRONLY|File::CREAT|File::EXCL, 0600) at_exit { File.unlink(lock_path) rescue nil } rescue Errno::EEXIST sock.close rescue nil retry end sock.close rescue nil port end ----------------------------- 8< -------------------------- I should probably take out the project-specific "xxxxxx_" prefix here: lock_path = "#{Dir::tmpdir}/test.#{addr}:#{port}.lock" Or even spinning that into its own gem since I expect to have more projects using it... > git://github.com/tarcieri/rev.git > > You will need to "rake gem" in order to build the gem. After that you can > install it. I had to manually build README to get "rake gem" to work via "cp README.textile README" -- Eric Wong From normalperson at yhbt.net Thu Aug 27 16:45:59 2009 From: normalperson at yhbt.net (Eric Wong) Date: Thu, 27 Aug 2009 13:45:59 -0700 Subject: [Rev-talk] Prepping for rev 0.3.0 In-Reply-To: <20090827201202.GB3185@dcvr.yhbt.net> References: <20090827201202.GB3185@dcvr.yhbt.net> Message-ID: <20090827204559.GC3185@dcvr.yhbt.net> Eric Wong wrote: > Tony Arcieri wrote: > > - Uses iobuffer gem instead of its own Rev::Buffer class. This gem is now > > optimized so it no longer makes any system calls aside from read/write, > > whereas the previous one needlessly polled for the current time in many > > cases (as part of its memory pooling). > > I just tried installing iobuffer under 1.8.7 and got this: > > cc -I. -I. -I/home/ew/lib/ruby/1.8/x86_64-linux -I. -fPIC -pipe -Wall -O2 -momit-leaf-frame-pointer -c iobuffer.c > iobuffer.c: In function ?IO_Buffer_read_from?: > iobuffer.c:292: error: ?rb_io_t? has no member named ?fd? > iobuffer.c: In function ?IO_Buffer_write_to?: > iobuffer.c:315: error: ?rb_io_t? has no member named ?fd? > make: *** [iobuffer.o] Error 1 > > It installs fine under 1.8.6 and 1.9.1. Unfortunately many people got > switched to 1.8.7 by their vendor... Not sure if you care. I've also pushed out the following to git://yhbt.net/iobuffer which works for me. Is there a separate mailing list for iobuffer? >From 5ad18571d436e1d84902b42dcd7690eb0d3d93e8 Mon Sep 17 00:00:00 2001 From: Eric Wong Date: Thu, 27 Aug 2009 13:40:09 -0700 Subject: [PATCH] Fix build with Ruby 1.8.7 Also tested with 1.8.6 and 1.9.1 Signed-off-by: Eric Wong --- ext/extconf.rb | 3 +++ ext/iobuffer.c | 2 +- 2 files changed, 4 insertions(+), 1 deletions(-) diff --git a/ext/extconf.rb b/ext/extconf.rb index 41706b7..7219989 100644 --- a/ext/extconf.rb +++ b/ext/extconf.rb @@ -2,5 +2,8 @@ require 'mkmf' dir_config("iobuffer") have_library("c", "main") +if have_macro("HAVE_RB_IO_T", "rubyio.h") + have_struct_member("rb_io_t", "fd", "rubyio.h") +end create_makefile("iobuffer") diff --git a/ext/iobuffer.c b/ext/iobuffer.c index 1485e04..30f798d 100644 --- a/ext/iobuffer.c +++ b/ext/iobuffer.c @@ -15,7 +15,7 @@ #include /* Macro for retrieving the file descriptor from an FPTR */ -#if !HAVE_RB_IO_T || (RUBY_VERSION_MAJOR == 1 && RUBY_VERSION_MINOR == 8) +#if !HAVE_RB_IO_T_FD #define FPTR_TO_FD(fptr) fileno(fptr->f) #else #define FPTR_TO_FD(fptr) fptr->fd -- Eric Wong From normalperson at yhbt.net Thu Aug 27 17:50:38 2009 From: normalperson at yhbt.net (Eric Wong) Date: Thu, 27 Aug 2009 14:50:38 -0700 Subject: [Rev-talk] Prepping for rev 0.3.0 In-Reply-To: References: <20090827201202.GB3185@dcvr.yhbt.net> Message-ID: <20090827215038.GA11313@dcvr.yhbt.net> Tony Arcieri wrote: > On Thu, Aug 27, 2009 at 2:12 PM, Eric Wong wrote: > > Tony Arcieri wrote: > > > I am preparing to release rev 0.3.0 with the following changes: > > > > > - Added Rev::Listener#fileno for accessing the underlying file descriptor > > > > > > - Support for creating Rev::Listeners from existing > > TCPServers/UNIXServers > > > > Cool. I've just pushed out similar changes to Rev::Server that I may > > need in the near future: > > > > Eric Wong (2): > > Rev::UNIXServer: use path instead of the first argument > > Rev::Server-based classes can build off ::*Server objects > > > > lib/rev/server.rb | 21 ++++++++++++++++++--- > > spec/unix_server_spec.rb | 25 +++++++++++++++++++++++++ > > 2 files changed, 43 insertions(+), 3 deletions(-) > > > > Pullable from git://yhbt.net/rev.git > > (browsable from http://git.bogomips.org/cgit/rev.git, too) > > Yes, those items are your changes merged into trunk. Guess you missed my > email when I said I did that. No, the original ones were for Rev::Listener, these are for Rev::Server. I just made these today :) > > I had to manually build README to get "rake gem" to work via > > "cp README.textile README" > > Weird, I thought I had all of those taken care of. What was the error you > encountered? Nevermind, works fine now. I think I was on a different machine after I pulled from you on one :) -- Eric Wong From normalperson at yhbt.net Sat Aug 29 01:13:45 2009 From: normalperson at yhbt.net (Eric Wong) Date: Fri, 28 Aug 2009 22:13:45 -0700 Subject: [Rev-talk] Prepping for rev 0.3.0 In-Reply-To: References: <20090827201202.GB3185@dcvr.yhbt.net> <20090827215038.GA11313@dcvr.yhbt.net> Message-ID: <20090829051345.GB17495@dcvr.yhbt.net> Tony Arcieri wrote: > Eric, > > I've merged your changes to Rev and iobuffer, and released a new iobuffer > gem. Let me know if everything looks good. Hi Tony, Everything looks good, thanks! -- Eric Wong