From normalperson at yhbt.net Sat Oct 3 16:19:57 2009 From: normalperson at yhbt.net (Eric Wong) Date: Sat, 3 Oct 2009 13:19:57 -0700 Subject: [Revactor-talk] [PATCH] swap Rev::Buffer for IO::Buffer Message-ID: <20091003201957.GA23085@dcvr.yhbt.net> Newer versions of Rev depend on a buffer implementation in a package outside of Rev. --- I've pushed this out to git://yhbt.net/revactor.git as well, only lightly tested but seems to work. lib/revactor/filters/packet.rb | 2 +- lib/revactor/tcp.rb | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/lib/revactor/filters/packet.rb b/lib/revactor/filters/packet.rb index d0b5da6..3f51c59 100644 --- a/lib/revactor/filters/packet.rb +++ b/lib/revactor/filters/packet.rb @@ -23,7 +23,7 @@ module Revactor @data_size = 0 @mode = :prefix - @buffer = Rev::Buffer.new + @buffer = IO::Buffer.new end # Callback for processing incoming frames diff --git a/lib/revactor/tcp.rb b/lib/revactor/tcp.rb index 3ae027d..d87a8a1 100644 --- a/lib/revactor/tcp.rb +++ b/lib/revactor/tcp.rb @@ -102,7 +102,7 @@ module Revactor @filterset ||= [*initialize_filter(options[:filter])] @receiver = @controller - @read_buffer = Rev::Buffer.new + @read_buffer = IO::Buffer.new end def inspect -- Eric Wong From normalperson at yhbt.net Sun Oct 4 04:35:06 2009 From: normalperson at yhbt.net (Eric Wong) Date: Sun, 4 Oct 2009 01:35:06 -0700 Subject: [Revactor-talk] [PATCH] support UNIX domain socket servers and client Message-ID: <20091004083506.GA8950@dcvr.yhbt.net> This is heavily-based on the existing TCP implementation but seems to work with the limited testing I've done. --- Also pushed out to git://yhbt.net/revactor as usual This ork is for a new experimental HTTP server capable of listening on UNIX domain sockets called "Rainbows!". It'll feature Revactor as one of the supported concurrency models ("rainbows" refers to the diversity of concurrency models supported :). The project is still very much in its infancy, but you can start reading about it here: http://rainbows.rubyforge.org/ and clone here git://git.bogomips.org/rainbows.git This also uses some of the Rev work I did earlier to allow Rev/Revactor to attach itself to existing core Ruby *Socket objects[1]. [1] there is/was also a rainbows branch of Unicorn (the server Rainbows! is based on) a while back but I decided to split rainbows off into a separate project since its very experimental and new-fangled concurrency things make some folks uncomfortable. lib/revactor.rb | 3 +- lib/revactor/unix.rb | 400 ++++++++++++++++++++++++++++++++++++++++++++++++++ spec/unix_spec.rb | 65 ++++++++ 3 files changed, 467 insertions(+), 1 deletions(-) create mode 100644 lib/revactor/unix.rb create mode 100644 spec/unix_spec.rb diff --git a/lib/revactor.rb b/lib/revactor.rb index f22b3a9..105fd70 100644 --- a/lib/revactor.rb +++ b/lib/revactor.rb @@ -32,7 +32,7 @@ module Revactor end %w{ - actor scheduler mailbox tcp http_client + actor scheduler mailbox tcp unix http_client filters/line filters/packet actorize }.each do |file| require File.dirname(__FILE__) + '/revactor/' + file @@ -41,6 +41,7 @@ end # Place Revactor modules and classes under the Actor namespace class Actor Actor::TCP = Revactor::TCP unless defined? Actor::TCP + Actor::UNIX = Revactor::UNIX unless defined? Actor::UNIX Actor::Filter = Revactor::Filter unless defined? Actor::Filter Actor::HttpClient = Revactor::HttpClient unless defined? Actor::HttpClient end diff --git a/lib/revactor/unix.rb b/lib/revactor/unix.rb new file mode 100644 index 0000000..5e81854 --- /dev/null +++ b/lib/revactor/unix.rb @@ -0,0 +1,400 @@ +#-- +# Copyright (C)2009 Eric Wong +# You can redistribute this under the terms of the Ruby license +# See file LICENSE for details +#++ + +module Revactor + # The UNIX module holds all Revactor functionality related to the + # UNIX domain sockets, including drop-in replacements + # for Ruby UNIX Sockets which can operate concurrently using Actors. + module UNIX + # Number of seconds to wait for a connection + CONNECT_TIMEOUT = 10 + + # Raised when a connection to a server fails + class ConnectError < StandardError; end + + # Connect to the specified path for a UNIX domain socket + # Accepts the following options: + # + # :active - Controls how data is read from the socket. See the + # documentation for Revactor::UNIX::Socket#active= + # + def self.connect(path, options = {}) + socket = begin + Socket.connect path, options + rescue SystemCallError + raise ConnectError, "connection refused" + end + socket.attach Rev::Loop.default + end + + # Listen on the specified path. Accepts the following options: + # + # :active - Default active setting for new connections. See the + # documentation Rev::UNIX::Socket#active= for more info + # + # :controller - The controlling actor, default Actor.current + # + # :filter - An symbol/class or array of symbols/classes which implement + # #encode and #decode methods to transform data sent and + # received data respectively via Revactor::UNIX::Socket. + # See the "Filters" section in the README for more information + # + def self.listen(path, options = {}) + Listener.new(path, options).attach(Rev::Loop.default).disable + end + + # UNIX socket class, returned by Revactor::UNIX.connect and + # Revactor::UNIX::Listener#accept + class Socket < Rev::UNIXSocket + attr_reader :controller + + class << self + # Connect to the specified path. Accepts the following options: + # + # :active - Controls how data is read from the socket. See the + # documentation for #active= + # + # :controller - The controlling actor, default Actor.current + # + # :filter - An symbol/class or array of symbols/classes which + # implement #encode and #decode methods to transform + # data sent and received data respectively via + # Revactor::UNIX::Socket. See the "Filters" section + # in the README for more information + # + def connect(path, options = {}) + options[:active] ||= false + options[:controller] ||= Actor.current + + super.instance_eval { + @active, @controller = options[:active], options[:controller] + @filterset = [*initialize_filter(options[:filter])] + self + } + end + end + + def initialize(socket, options = {}) + super(socket) + + @active ||= options[:active] || false + @controller ||= options[:controller] || Actor.current + @filterset ||= [*initialize_filter(options[:filter])] + + @receiver = @controller + @read_buffer = IO::Buffer.new + end + + def inspect + "#<#{self.class}:0x#{object_id.to_s(16)} #@address_family:#@path" + end + + # Enable or disable active mode data reception. State can be any + # of the following: + # + # true - All received data is sent to the controlling actor + # false - Receiving data is disabled + # :once - A single message will be sent to the controlling actor + # then active mode will be disabled + # + def active=(state) + unless @receiver == @controller + raise "cannot change active state during a synchronous call" + end + + unless [true, false, :once].include? state + raise ArgumentError, "must be true, false, or :once" + end + + if [true, :once].include?(state) + unless @read_buffer.empty? + @receiver << [:unix, self, @read_buffer.read] + return if state == :once + end + + enable unless enabled? + end + + @active = state + end + + # Is the socket in active mode? + def active?; @active; end + + # Set the controlling actor + def controller=(controller) + Actor === controller or + raise ArgumentError, "controller must be an actor" + + @receiver = controller if @receiver == @controller + @controller = controller + end + + # Read data from the socket synchronously. If a length is specified + # then the call blocks until the given length has been read. Otherwise + # the call blocks until it receives any data. + def read(length = nil) + # Only one synchronous call allowed at a time + @receiver == @controller or + raise "already being called synchronously" + + unless @read_buffer.empty? or (length and @read_buffer.size < length) + return @read_buffer.read(length) + end + + active = @active + @active = :once + @receiver = Actor.current + enable unless enabled? + + loop do + Actor.receive do |filter| + filter.when(T[:unix, self]) do |_, _, data| + if length.nil? + @receiver = @controller + @active = active + enable if @active + + return data + end + + @read_buffer << data + + if @read_buffer.size >= length + @receiver = @controller + @active = active + enable if @active + + return @read_buffer.read(length) + end + end + + filter.when(T[:unix_closed, self]) do + unless @receiver == @controller + @receiver = @controller + @receiver << T[:unix_closed, self] + end + + raise EOFError, "connection closed" + end + end + end + end + + # Write data to the socket. The call blocks until all data has been + # written. + def write(data) + # Only one synchronous call allowed at a time + @receiver == @controller or + raise "already being called synchronously" + + active = @active + @active = false + @receiver = Actor.current + disable if @active + + super(encode(data)) + + Actor.receive do |filter| + filter.when(T[:unix_write_complete, self]) do + @receiver = @controller + @active = active + enable if @active and not enabled? + + return data.size + end + + filter.when(T[:unix_closed, self]) do + @active = false + raise EOFError, "connection closed" + end + end + end + + alias_method :<<, :write + + ######### + protected + ######### + + # + # Filter setup + # + + # Initialize filters + def initialize_filter(filter) + case filter + when NilClass + [] + when Tuple + name, *args = filter + case name + when Class + name.new(*args) + when Symbol + symbol_to_filter(name).new(*args) + else raise ArgumentError, "unrecognized filter type: #{name.class}" + end + when Array + filter.map { |f| initialize_filter f } + when Class + filter.new + when Symbol + symbol_to_filter(filter).new + end + end + + # Lookup filters referenced as symbols + def symbol_to_filter(filter) + case filter + when :line then Revactor::Filter::Line + when :packet then Revactor::Filter::Packet + else raise ArgumentError, "unrecognized filter type: #{filter}" + end + end + + # Decode data through the filter chain + def decode(data) + @filterset.inject([data]) do |a, filter| + a.inject([]) do |a2, d| + a2 + filter.decode(d) + end + end + end + + # Encode data through the filter chain + def encode(message) + @filterset.reverse.inject(message) { |m, filter| filter.encode(*m) } + end + + # + # Rev::UNIXSocket callback + # + + def on_connect + @receiver << T[:unix_connected, self] + end + + def on_connect_failed + @receiver << T[:unix_connect_failed, self] + end + + def on_close + @receiver << T[:unix_closed, self] + end + + def on_read(data) + # Run incoming message through the filter chain + message = decode(data) + + if message.is_a?(Array) and not message.empty? + message.each { |msg| @receiver << T[:unix, self, msg] } + elsif message and not message.empty? + @receiver << T[:unix, self, message] + else return + end + + if @active == :once + @active = false + disable + end + end + + def on_write_complete + @receiver << T[:unix_write_complete, self] + end + end + + # UNIX Listener returned from Revactor::UNIX.listen + class Listener < Rev::UNIXListener + attr_reader :controller + + # Listen on the specified path. Accepts the following options: + # + # :active - Default active setting for new connections. See the + # documentation Rev::UNIX::Socket#active= for more info + # + # :controller - The controlling actor, default Actor.current + # + # :filter - An symbol/class or array of symbols/classes which implement + # #encode and #decode methods to transform data sent and + # received data respectively via Revactor::UNIX::Socket. + # See the "Filters" section in the README for more information + # + def initialize(path, options = {}) + super(path) + opts = { + :active => false, + :controller => Actor.current + }.merge(options) + + @active, @controller = opts[:active], opts[:controller] + @filterset = options[:filter] + + @accepting = false + end + + def inspect + "#<#{self.class}:0x#{object_id.to_s(16)}>" + end + + # Change the default active setting for newly accepted connections + def active=(state) + unless [true, false, :once].include? state + raise ArgumentError, "must be true, false, or :once" + end + + @active = state + end + + # Will newly accepted connections be active? + def active?; @active; end + + # Change the default controller for newly accepted connections + def controller=(controller) + Actor === controller or + raise ArgumentError, "controller must be an actor" + @controller = controller + end + + # Accept an incoming connection + def accept + raise "another actor is already accepting" if @accepting + + @accepting = true + @receiver = Actor.current + enable + + Actor.receive do |filter| + filter.when(T[:unix_connection, self]) do |_, _, sock| + @accepting = false + return sock + end + end + end + + ######### + protected + ######### + + # + # Rev::UNIXListener callbacks + # + + def on_connection(socket) + sock = Socket.new(socket, + :controller => @controller, + :active => @active, + :filter => @filterset + ) + sock.attach(evloop) + + @receiver << T[:unix_connection, self, sock] + disable + end + end + end +end diff --git a/spec/unix_spec.rb b/spec/unix_spec.rb new file mode 100644 index 0000000..4bda4b4 --- /dev/null +++ b/spec/unix_spec.rb @@ -0,0 +1,65 @@ +#-- +# Copyright (C)2009 Eric Wong +# You can redistribute this under the terms of the Ruby license +# See file LICENSE for details +#++ + +require File.dirname(__FILE__) + '/../lib/revactor' +require 'tempfile' + +describe Revactor::UNIX do + before :each do + @actor_run = false + @tmp = Tempfile.new('unix.sock') + File.unlink(@tmp.path) + @server = UNIXServer.new(@tmp.path) + end + + after :each do + @server.close unless @server.closed? + File.unlink(@tmp.path) + end + + it "connects to remote servers" do + sock = Revactor::UNIX.connect(@tmp.path) + sock.should be_an_instance_of(Revactor::UNIX::Socket) + @server.accept.should be_an_instance_of(UNIXSocket) + + sock.close + end + + it "listens for remote connections" do + # Don't use their server for this one... + @server.close + File.unlink(@tmp.path) + + server = Revactor::UNIX.listen(@tmp.path) + server.should be_an_instance_of(Revactor::UNIX::Listener) + + s1 = UNIXSocket.open(@tmp.path) + s2 = server.accept + + server.close + s2.close + end + + it "reads data" do + s1 = Revactor::UNIX.connect(@tmp.path) + s2 = @server.accept + + s2.write 'foobar' + s1.read(6).should == 'foobar' + + s1.close + end + + it "writes data" do + s1 = Revactor::UNIX.connect(@tmp.path) + s2 = @server.accept + + s1.write 'foobar' + s2.read(6).should == 'foobar' + + s1.close + end +end -- Eric Wong From normalperson at yhbt.net Mon Oct 5 14:24:47 2009 From: normalperson at yhbt.net (Eric Wong) Date: Mon, 5 Oct 2009 11:24:47 -0700 Subject: [Revactor-talk] [PATCH] support UNIX domain socket servers and client In-Reply-To: References: <20091004083506.GA8950@dcvr.yhbt.net> Message-ID: <20091005182447.GA25569@dcvr.yhbt.net> Tony Arcieri wrote: > Merged and pushed. > > I'll do a release here shortly unless you have additional patches to push. Nope, that's it for now :) Thanks! Hopefully I'll get more time to actually play with (and pound the crap out of) Revactor+Rainbows! this week :) -- Eric Wong From normalperson at yhbt.net Sun Oct 11 02:44:55 2009 From: normalperson at yhbt.net (Eric Wong) Date: Sat, 10 Oct 2009 23:44:55 -0700 Subject: [Revactor-talk] [PATCH/RFC] TCP: allow optional :timeout arguments in a hash Message-ID: <20091011064455.GA16264@dcvr.yhbt.net> When dealing with untrusted/unreliable clients, it can be dangerous to tie up server resources with slowly-responding client connections. So implement optional timeouts that may be used on a per-call basis. Using an options hash here to allow for future features to be implemented more easily. --- I'm not sure if Revactor::TCP is the correct namespace for WriteError and ReadError exceptions, since I'd also like timeouts for Revactor::UNIX, mainly for API reusability All the read/write-using code in Rainbows! uses the exact same code for UNIX and TCP sockets. I've pushed out to git://yhbt.net/revactor for review, but I don't think this should be put into a stable release just yet... lib/revactor/tcp.rb | 18 ++++++++++++++++-- spec/tcp_spec.rb | 23 +++++++++++++++++++++++ 2 files changed, 39 insertions(+), 2 deletions(-) diff --git a/lib/revactor/tcp.rb b/lib/revactor/tcp.rb index d87a8a1..e52b792 100644 --- a/lib/revactor/tcp.rb +++ b/lib/revactor/tcp.rb @@ -12,6 +12,12 @@ module Revactor # Number of seconds to wait for a connection CONNECT_TIMEOUT = 10 + # Raised when a read from a client or server fails + class ReadError < StandardError; end + + # Raised when a write to a client or server fails + class WriteError < StandardError; end + # Raised when a connection to a remote server fails class ConnectError < StandardError; end @@ -152,7 +158,7 @@ module Revactor # Read data from the socket synchronously. If a length is specified # then the call blocks until the given length has been read. Otherwise # the call blocks until it receives any data. - def read(length = nil) + def read(length = nil, options = {}) # Only one synchronous call allowed at a time raise "already being called synchronously" unless @receiver == @controller @@ -195,12 +201,16 @@ module Revactor raise EOFError, "connection closed" end + + if timeout = options[:timeout] + filter.after(timeout) { raise ReadError, "read timed out" } + end end end end # Write data to the socket. The call blocks until all data has been written. - def write(data) + def write(data, options = {}) # Only one synchronous call allowed at a time raise "already being called synchronously" unless @receiver == @controller @@ -224,6 +234,10 @@ module Revactor @active = false raise EOFError, "connection closed" end + + if timeout = options[:timeout] + filter.after(timeout) { raise WriteError, "write timed out" } + end end end diff --git a/spec/tcp_spec.rb b/spec/tcp_spec.rb index 50f2940..7e02653 100644 --- a/spec/tcp_spec.rb +++ b/spec/tcp_spec.rb @@ -52,6 +52,29 @@ describe Revactor::TCP do s1.close end + it "times out on read" do + s1 = Revactor::TCP.connect(TEST_HOST, RANDOM_PORT) + s2 = @server.accept + + proc { + s1.read(6, :timeout => 0.1) + }.should raise_error(Revactor::TCP::ReadError) + + s1.close + end + + it "times out on write" do + s1 = Revactor::TCP.connect(TEST_HOST, RANDOM_PORT) + s2 = @server.accept + + # typically needs to write several thousand kilobytes before it blocks + buf = ' ' * 16384 + proc { + loop { s1.write(buf, :timeout => 0.1) } + }.should raise_error(Revactor::TCP::WriteError) + s1.close + end + it "writes data" do s1 = Revactor::TCP.connect(TEST_HOST, RANDOM_PORT) s2 = @server.accept -- Eric Wong From normalperson at yhbt.net Mon Oct 19 14:58:44 2009 From: normalperson at yhbt.net (Eric Wong) Date: Mon, 19 Oct 2009 11:58:44 -0700 Subject: [Revactor-talk] [ANN] Rainbows! HTTP server powered by Revactor (or Rev) Message-ID: <20091019185844.GD11442@dcvr.yhbt.net> Hi all, Just wanted to make sure that fans of Revactor and/or Rev know there's a new HTTP server that can use either Revactor or Rev (without Revactor) as it's network concurrency backend[1]. It's based on Unicorn so it supports the Rack specification and uses Unix-only preforking[2] + signal management. Linux 2.6+ is recommended for epoll support, but it should work for kqueue users since it's all based on libev. Feedback from users of all systems supported by Rev would be greatly appreciated. On a related note: Revactor has been really nice to work with, hopefully this can help increase 1.9 adoption :) [1] - it also supports two threading models that can be loaded independently of Rev/Revactor and will eventually support EventMachine, too. [2] - local concurrency != remote concurrency, (this really is the whole point of Unicorn + Rainbows!). Preforking is to handle the former, Rev/Revactor is for the latter. Links: * http://rainbows.rubyforge.org/ * rainbows-talk at rubyforge.org * git://git.bogomips.org/rainbows.git This will be my first/last announcement to {rev,revactor}-talk lists, but of course I'll post any improvements I have for rev/revactor on here in the future :> -- Eric Wong From ibc at aliax.net Mon Oct 19 16:52:01 2009 From: ibc at aliax.net (=?iso-8859-1?q?I=F1aki_Baz_Castillo?=) Date: Mon, 19 Oct 2009 22:52:01 +0200 Subject: [Revactor-talk] [ANN] Rainbows! HTTP server powered by Revactor (or Rev) In-Reply-To: <20091019185844.GD11442@dcvr.yhbt.net> References: <20091019185844.GD11442@dcvr.yhbt.net> Message-ID: <200910192252.02049.ibc@aliax.net> El Lunes, 19 de Octubre de 2009, Eric Wong escribi?: > Hi all, > > Just wanted to make sure that fans of Revactor and/or Rev know there's a > new HTTP server that can use either Revactor or Rev (without Revactor) > as it's network concurrency backend[1]. > > It's based on Unicorn so it supports the Rack specification and uses > Unix-only preforking[2] + signal management. Linux 2.6+ is recommended > for epoll support, but it should work for kqueue users since it's all > based on libev. Feedback from users of all systems supported by Rev > would be greatly appreciated. > > > On a related note: Revactor has been really nice to work with, hopefully > this can help increase 1.9 adoption :) > > > [1] - it also supports two threading models that can be loaded > independently of Rev/Revactor and will eventually support > EventMachine, too. > > [2] - local concurrency != remote concurrency, (this really is > the whole point of Unicorn + Rainbows!). Preforking is > to handle the former, Rev/Revactor is for the latter. Great. Is there any performance comparission with other Rack web servers as Thin or Ebb? I'm building right now a Rack application and choosing the web server :) -- I?aki Baz Castillo From normalperson at yhbt.net Mon Oct 19 17:21:18 2009 From: normalperson at yhbt.net (Eric Wong) Date: Mon, 19 Oct 2009 14:21:18 -0700 Subject: [Revactor-talk] [ANN] Rainbows! HTTP server powered by Revactor (or Rev) In-Reply-To: <200910192252.02049.ibc@aliax.net> References: <20091019185844.GD11442@dcvr.yhbt.net> <200910192252.02049.ibc@aliax.net> Message-ID: <20091019212118.GA17400@dcvr.yhbt.net> I?aki Baz Castillo wrote: > El Lunes, 19 de Octubre de 2009, Eric Wong escribi?: > > Hi all, > > [2] - local concurrency != remote concurrency, (this really is > > the whole point of Unicorn + Rainbows!). Preforking is > > to handle the former, Rev/Revactor is for the latter. > > Great. Is there any performance comparission with other Rack web servers as > Thin or Ebb? No, not yet, I also added this blurb in http://rainbows.rubyforge.org/TUNING.html earlier today: As of October 2009, absolutely ZERO work has been done for performance validation and tuning. Furthermore, \Rainbows! is NOT expected to do well on traditional benchmarks. Remember that \Rainbows! is only designed for applications that sleep and/or trickle network traffic. In the future, *may* do well in traditional benchmarks as a side effect, but that will never be the primary goal of the project. I also don't believe it's good form for me to publish comparisons against other web servers, so I have no plans to do that myself. > I'm building right now a Rack application and choosing the web server :) If it's a more traditional application with few/no external dependencies, Unicorn seems to be pretty solid by now and some people have started using it. Unicorn just requires nginx in front if you're serving non-local traffic. I've intentionally kept the Unicorn <=> Rainbows! migration paths simple and bidirectional so it's easy to switch between them depending on your needs. -- Eric Wong From tony at medioh.com Tue Oct 27 01:18:06 2009 From: tony at medioh.com (Tony Arcieri) Date: Mon, 26 Oct 2009 23:18:06 -0600 Subject: [Revactor-talk] Fwd: [ANN] Revactor 0.1.5: Erlangy add-ons to Ruby 1.9 In-Reply-To: References: Message-ID: *"The reports of my death are greatly exaggerated"* Revactor implements an approach to concurrency known as the Actor model. Actors are lightweight concurrency primitives which talk to each other using messages. While it can't provide you the full benefits of Erlang's immutable state and excellent performance, Revactor can provide an easier way of modeling complex I/O problems in Ruby that might be difficult to think about when forced into the inversion of control model used by standard asynchronous event frameworks, such as EventMachine or Rev. Revactor uses the Ruby 1.9 Fibers mechanism to provide (relatively) high performance userspace concurrency for I/O bound tasks. This makes it useful for a number of tasks which are presently somewhat difficult to do in Ruby, such as web spidering and asynchronous web frameworks. Revactor's actor API is loosely shared with two other Ruby projects: the Rubinius virtual machine's actor library (http://rubini.us/) and the Omnibus Concurrency Library (http://rubyforge.org/projects/concurrent/), both authored by MenTaLguY. The syntax of these implementations and Revactor are mostly equivalent, however semantics of these three APIs are only loosely similar and some work will be involved in porting code between them. -- Revactor 0.1.5 is a maintenance release to provide compatibility with Rev 0.3.x, a library it uses for event monitoring. However, it also includes a new feature: Revactor::HttpFetcher provides a concurrent HTTP fetcher library useful for tasks like web spidering, or pretty much any situation where you need to fetch a lot of documents over HTTP and might discover new URLs along the way. This fetcher uses a scatter/gather approach to spreading a fetch queue among as many fibers as you wish, and allows you to continuously add URLs to the fetch queue as they enter the system. You can view the implementation/unprocessed RDoc here: http://github.com/tarcieri/revactor/blob/master/lib/revactor/http_fetcher.rb In production, a single process on a 2.0GHz machine was able to pull down around 10Mbps worth of documents. YMMV, but I would be interested to hear how it performs for other people. Also, check out Rainbows, which uses Revactor and adds async support to the Unicorn HTTP server, and provides something like the Python Tornado HTTP server (http://www.tornadoweb.org/) in Ruby: http://rainbows.rubyforge.org/ -- Revactor Philosophy: http://revactor.org/philosophy Github: http://github.com/tarcieri/revactor Gemcutter: http://gemcutter.org/gems/revactor Rubyforge: http://rubyforge.org/projects/revactor/ -- Tony Arcieri -------------- next part -------------- An HTML attachment was scrubbed... URL: From normalperson at yhbt.net Tue Oct 27 05:10:26 2009 From: normalperson at yhbt.net (Eric Wong) Date: Tue, 27 Oct 2009 02:10:26 -0700 Subject: [Revactor-talk] Fwd: [ANN] Revactor 0.1.5: Erlangy add-ons to Ruby 1.9 In-Reply-To: References: Message-ID: <20091027091026.GE29122@dcvr.yhbt.net> Tony Arcieri wrote: > Also, check out Rainbows, which uses Revactor and adds async support to the > Unicorn HTTP server, and provides something like the Python Tornado HTTP > server (http://www.tornadoweb.org/) in Ruby: > > http://rainbows.rubyforge.org/ Thanks for the plug, and perfect timing too. I got Rainbows! 0.4.0 to depend on Revactor 0.1.5 for the Unix socket support. It'll be interesting to see how the various Rainbows! concurrency models compare on real applications, but I'm personally favoring Revactor :) -- Eric Wong