From normalperson at yhbt.net Fri Nov 27 01:49:37 2009 From: normalperson at yhbt.net (Eric Wong) Date: Thu, 26 Nov 2009 22:49:37 -0800 Subject: [Rev-talk] [PATCH 2/2] Make Rev::Loop#run_once less intrusive with 1.8 threads In-Reply-To: <1259304577-6055-1-git-send-email-normalperson@yhbt.net> References: <1259304577-6055-1-git-send-email-normalperson@yhbt.net> Message-ID: <1259304577-6055-3-git-send-email-normalperson@yhbt.net> Instead of blocking for 10ms when other threads are present, run in non-blocking mode if we're not the only thread in the process. This change does not negatively impact 1.9. --- ext/rev/extconf.rb | 4 ++++ ext/rev/rev_loop.c | 37 ++++++++++++++++++++++++++----------- 2 files changed, 30 insertions(+), 11 deletions(-) diff --git a/ext/rev/extconf.rb b/ext/rev/extconf.rb index 9b48dd9..2422cbc 100644 --- a/ext/rev/extconf.rb +++ b/ext/rev/extconf.rb @@ -8,6 +8,10 @@ if have_func('rb_thread_blocking_region') $defs << '-DHAVE_RB_THREAD_BLOCKING_REGION' end +if have_func('rb_thread_alone') + $defs << '-DHAVE_RB_THREAD_ALONE' +end + if have_func('rb_str_set_len') $defs << '-DHAVE_RB_STR_SET_LEN' end diff --git a/ext/rev/rev_loop.c b/ext/rev/rev_loop.c index 86dad37..bde0021 100644 --- a/ext/rev/rev_loop.c +++ b/ext/rev/rev_loop.c @@ -12,6 +12,14 @@ #include "rev.h" +#if defined(HAVE_RB_THREAD_BLOCKING_REGION) +# define Rev_Loop_may_block_safely() (1) +#elif defined(HAVE_RB_THREAD_ALONE) +# define Rev_Loop_may_block_safely() (rb_thread_alone()) +#else /* just in case Ruby changes: */ +# define Rev_Loop_may_block_safely() (0) +#endif + static VALUE mRev = Qnil; static VALUE cRev_Loop = Qnil; @@ -175,19 +183,26 @@ void Rev_Loop_process_event(VALUE watcher, int revents) */ static VALUE Rev_Loop_run_once(VALUE self) { - struct Rev_Loop *loop_data; VALUE nevents; - - Data_Get_Struct(self, struct Rev_Loop, loop_data); - assert(loop_data->ev_loop && !loop_data->events_received); - - Rev_Loop_ev_loop_oneshot(loop_data); - Rev_Loop_dispatch_events(loop_data); - - nevents = INT2NUM(loop_data->events_received); - loop_data->events_received = 0; - + if (Rev_Loop_may_block_safely()) { + struct Rev_Loop *loop_data; + + Data_Get_Struct(self, struct Rev_Loop, loop_data); + + assert(loop_data->ev_loop && !loop_data->events_received); + + Rev_Loop_ev_loop_oneshot(loop_data); + Rev_Loop_dispatch_events(loop_data); + + nevents = INT2NUM(loop_data->events_received); + loop_data->events_received = 0; + + } else { + nevents = Rev_Loop_run_nonblock(self); + rb_thread_schedule(); + } + return nevents; } -- 1.6.5.3.300.g9be30 From normalperson at yhbt.net Fri Nov 27 01:49:36 2009 From: normalperson at yhbt.net (Eric Wong) Date: Thu, 26 Nov 2009 22:49:36 -0800 Subject: [Rev-talk] [PATCH 1/2] make Rev::Loop#run_nonblock signal-safe for 1.8 In-Reply-To: <1259304577-6055-1-git-send-email-normalperson@yhbt.net> References: <1259304577-6055-1-git-send-email-normalperson@yhbt.net> Message-ID: <1259304577-6055-2-git-send-email-normalperson@yhbt.net> Even though this method is currently not used anywhere, it may be useful in the next change(s) I make to make Rev + Threads run better under 1.8 --- ext/rev/rev_loop.c | 2 ++ 1 files changed, 2 insertions(+), 0 deletions(-) diff --git a/ext/rev/rev_loop.c b/ext/rev/rev_loop.c index 686e1a7..86dad37 100644 --- a/ext/rev/rev_loop.c +++ b/ext/rev/rev_loop.c @@ -260,7 +260,9 @@ static VALUE Rev_Loop_run_nonblock(VALUE self) assert(loop_data->ev_loop && !loop_data->events_received); + TRAP_BEG; RUN_LOOP(loop_data, EVLOOP_NONBLOCK); + TRAP_END; Rev_Loop_dispatch_events(loop_data); nevents = INT2NUM(loop_data->events_received); -- 1.6.5.3.300.g9be30 From normalperson at yhbt.net Fri Nov 27 01:49:35 2009 From: normalperson at yhbt.net (Eric Wong) Date: Thu, 26 Nov 2009 22:49:35 -0800 Subject: [Rev-talk] [PATCH 0/2] improve 1.8 performance when mixed with threads Message-ID: <1259304577-6055-1-git-send-email-normalperson@yhbt.net> Hi, Under 1.8, the RevThreadSpawn concurrency model of Rainbows! was performing horribly for processing large uploads. I narrowed it down to the blocking-for-10ms behavior of Rev under 1.8. Since the Thread running the Rev::Loop always gets scheduled, unconditionally blocking to do nothing for 10ms becomes very bad. Now, instead of blocking for 10ms unconditionally, we now detect if we're the lone thread in the process and run in non-blocking mode if there are any other threads. This change does not affect 1.9. With the current rainbows.git[1], the times for the following test goes from ~73 seconds to ~6 seconds. I'm not missing any zeros there :) # note(1), run "make -C t random_blob" first for more accurate timings # note(2), I don't use Rubygems for development, so setting up RUBYLIB # by hand to include rack, unicorn (and rev+iobuffer, of course) is # required make -C t models=RevThreadSpawn t0100-rack-input-hammer.sh The same test under 1.9 is around ~2.5 seconds. [1] 921a961e91276522c134813052a7b7e06b00ef2b git://git.bogomips.org/rainbows.git Both of these are pushed out to the usual location at git://yhbt.net/rev Eric Wong (2): make Rev::Loop#run_nonblock signal-safe for 1.8 Make Rev::Loop#run_once less intrusive with 1.8 threads From normalperson at yhbt.net Fri Nov 27 03:22:38 2009 From: normalperson at yhbt.net (Eric Wong) Date: Fri, 27 Nov 2009 00:22:38 -0800 Subject: [Rev-talk] [PATCH] async_watcher: do not trigger on spurious wakeups Message-ID: <20091127082238.GA14314@dcvr.yhbt.net> Also pushed out to git://yhbt.net/rev.git I actually found this bug while chasing down some weird behavior with IO#read under 1.8 (which does not affect 1.9). I was unable to reproduce what was happening inside Rainbows![1] in an isolated test case, but I found another failure case for this test. [1] - which does not share AsyncWatcher across fork, but does handle a lot of signals. >From e2ab2b774ce6431519a42ac5b831708f14069801 Mon Sep 17 00:00:00 2001 From: Eric Wong Date: Thu, 26 Nov 2009 18:11:17 -0800 Subject: [PATCH] async_watcher: do not trigger on spurious wakeups Also included is a convoluted test case that is able to reliably reproduce the failure on both a UP and SMP Linux 2.6 machines without this change. --- lib/rev/async_watcher.rb | 7 ++++- spec/async_watcher_spec.rb | 57 ++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 63 insertions(+), 1 deletions(-) create mode 100644 spec/async_watcher_spec.rb diff --git a/lib/rev/async_watcher.rb b/lib/rev/async_watcher.rb index d038742..df12c34 100644 --- a/lib/rev/async_watcher.rb +++ b/lib/rev/async_watcher.rb @@ -31,7 +31,12 @@ module Rev def on_readable # Read a byte from the pipe. This clears readability, unless # another signal is pending - @reader.read 1 + begin + @reader.read_nonblock 1 + rescue Errno::EAGAIN + # in case there are spurious wakeups from forked processs + return + end on_signal end end diff --git a/spec/async_watcher_spec.rb b/spec/async_watcher_spec.rb new file mode 100644 index 0000000..ecce516 --- /dev/null +++ b/spec/async_watcher_spec.rb @@ -0,0 +1,57 @@ +require File.dirname(__FILE__) + '/../lib/rev' +require 'tempfile' +require 'fcntl' + +describe Rev::AsyncWatcher do + + it "does not signal on spurious wakeups" do + aw = Rev::AsyncWatcher.new + tmp = Tempfile.new('rev_async_watcher_test') + nr_fork = 2 # must be at least two for spurious wakeups + + # We have aetter chance of failing if this overflows the pipe buffer + # which POSIX requires >= 512 bytes, Linux 2.6 uses 4096 bytes + nr_signal = 4096 * 4 + + append = File.open(tmp.path, "ab") + append.sync = true + rd, wr = ::IO.pipe + + aw.on_signal { append.syswrite("#$$\n") } + children = nr_fork.times.map do + fork do + trap(:TERM) { exit!(0) } + rloop = Rev::Loop.default + aw.attach(rloop) + wr.write '.' # signal to master that we're ready + rloop.run + exit!(1) # should not get here + end + end + + # ensure children are ready + nr_fork.times { rd.sysread(1).should == '.' } + + # send our signals + nr_signal.times { aw.signal } + + # wait for the pipe buffer to be consumed by the children + sleep 1 while tmp.stat.ctime >= (Time.now - 4) + + children.each do |pid| + Process.kill(:TERM, pid) + _, status = Process.waitpid2(pid) + status.exitstatus.should == 0 + end + + # we should've written a line for every signal we sent + lines = tmp.readlines + lines.size.should == nr_signal + + # theoretically a bad kernel scheduler could give us fewer... + lines.sort.uniq.size.should == nr_fork + + tmp.close! + end + +end -- Eric Wong From normalperson at yhbt.net Fri Nov 27 14:28:40 2009 From: normalperson at yhbt.net (Eric Wong) Date: Fri, 27 Nov 2009 11:28:40 -0800 Subject: [Rev-talk] [PATCH iobuffer] Allow global tuning of the default node size Message-ID: <20091127192840.GA16058@dcvr.yhbt.net> Hi, I've also pushed out to git://yhbt.net/iobuffer Background: Using the stupid dd(1)-like benchmark in Unicorn[1] with the Rainbows! Rev concurrency model, I noticed Rev being significantly[2] slower than the EventMachine one with certain response sizes, while other (both smaller and larger, the two were roughly on par[3]. It turned out the default benchmark defaults of writing 4096-byte response bodies managed to cause many extra allocations because of HTTP headers being written separately. Since Rev::IO::INPUT_SIZE is 16384, I figured it would make sense to use 16384 in Rainbows!, too. This should also have the pleasant side effect of allowing the malloc() implementation to share unused blocks between Rev input and output more easily. I noticed this performance improvement both using tcmalloc and glibc ptmalloc2. I would not hesitate changing the DEFAULT_NODE_SIZE to 16384 so applications are less likely to want to change it, too. [1] git://git.bogomips.org/unicorn.git test/benchmark/dd.ru [2] roughly half as fast as EM [3] EM is still a little faster, further investigation needed... >From 715cb9915de00e857f3e727e8c49117bbf06ffb7 Mon Sep 17 00:00:00 2001 From: Eric Wong Date: Fri, 27 Nov 2009 11:08:09 -0800 Subject: [PATCH] Allow global tuning of the default node size This is useful for applications that use Rev and do not allocate IO::Buffer objects directly. --- ext/iobuffer.c | 56 +++++++++++++++++++++++++++++++++++++++++++------- spec/buffer_spec.rb | 25 ++++++++++++++++++++++ 2 files changed, 73 insertions(+), 8 deletions(-) diff --git a/ext/iobuffer.c b/ext/iobuffer.c index 30f798d..aa8a6ab 100644 --- a/ext/iobuffer.c +++ b/ext/iobuffer.c @@ -23,6 +23,7 @@ /* Default number of bytes in each node's buffer. Should be >= MTU */ #define DEFAULT_NODE_SIZE 4096 +static unsigned default_node_size = DEFAULT_NODE_SIZE; struct buffer { unsigned size, node_size; @@ -43,6 +44,8 @@ static VALUE IO_Buffer_allocate(VALUE klass); static void IO_Buffer_mark(struct buffer *); static void IO_Buffer_free(struct buffer *); +static VALUE IO_Buffer_default_node_size(VALUE klass); +static VALUE IO_Buffer_set_default_node_size(VALUE klass, VALUE size); static VALUE IO_Buffer_initialize(int argc, VALUE *argv, VALUE self); static VALUE IO_Buffer_clear(VALUE self); static VALUE IO_Buffer_size(VALUE self); @@ -77,6 +80,11 @@ void Init_iobuffer() cIO_Buffer = rb_define_class_under(rb_cIO, "Buffer", rb_cObject); rb_define_alloc_func(cIO_Buffer, IO_Buffer_allocate); + rb_define_singleton_method(cIO_Buffer, "default_node_size", + IO_Buffer_default_node_size, 0); + rb_define_singleton_method(cIO_Buffer, "default_node_size=", + IO_Buffer_set_default_node_size, 1); + rb_define_method(cIO_Buffer, "initialize", IO_Buffer_initialize, -1); rb_define_method(cIO_Buffer, "clear", IO_Buffer_clear, 0); rb_define_method(cIO_Buffer, "size", IO_Buffer_size, 0); @@ -108,29 +116,61 @@ static void IO_Buffer_free(struct buffer *buf) } /** + * call-seq: + * IO_Buffer.default_node_size -> 4096 + * + * Retrieves the current value of the default node size. + */ +static VALUE IO_Buffer_default_node_size(VALUE klass) +{ + return UINT2NUM(default_node_size); +} + +/* + * safely converts node sizes from Ruby numerics to C and raising + * ArgumentError or RangeError on invalid sizes + */ +static unsigned convert_node_size(VALUE size) +{ + int node_size = NUM2INT(size); + + if(node_size < 1) rb_raise(rb_eArgError, "invalid buffer size"); + + return (unsigned)node_size; +} + +/** + * call-seq: + * IO_Buffer.default_node_size = 16384 + * + * Sets the default node size for calling IO::Buffer.new with no arguments. + */ +static VALUE IO_Buffer_set_default_node_size(VALUE klass, VALUE size) +{ + default_node_size = convert_node_size(size); + + return size; +} + +/** * call-seq: - * IO_Buffer.new(size = DEFAULT_NODE_SIZE) -> IO_Buffer + * IO_Buffer.new(size = IO::Buffer.default_node_size) -> IO_Buffer * * Create a new IO_Buffer with linked segments of the given size */ static VALUE IO_Buffer_initialize(int argc, VALUE *argv, VALUE self) { VALUE node_size_obj; - int node_size; struct buffer *buf; if(rb_scan_args(argc, argv, "01", &node_size_obj) == 1) { - node_size = NUM2INT(node_size_obj); - - if(node_size < 1) rb_raise(rb_eArgError, "invalid buffer size"); - Data_Get_Struct(self, struct buffer, buf); /* Make sure we're not changing the buffer size after data has been allocated */ assert(!buf->head); assert(!buf->pool_head); - buf->node_size = node_size; + buf->node_size = convert_node_size(node_size_obj); } return Qnil; @@ -329,7 +369,7 @@ static struct buffer *buffer_new(void) buf = (struct buffer *)xmalloc(sizeof(struct buffer)); buf->head = buf->tail = buf->pool_head = buf->pool_tail = 0; buf->size = 0; - buf->node_size = DEFAULT_NODE_SIZE; + buf->node_size = default_node_size; return buf; } diff --git a/spec/buffer_spec.rb b/spec/buffer_spec.rb index 4329bf6..fa38740 100644 --- a/spec/buffer_spec.rb +++ b/spec/buffer_spec.rb @@ -98,6 +98,31 @@ describe IO::Buffer do @buffer.should_not be_empty end + it "can set default node size" do + IO::Buffer.default_node_size = 1 + IO::Buffer.default_node_size.should == 1 + (IO::Buffer.default_node_size = 4096).should == 4096 + end + + it "can be created with a different node size" do + IO::Buffer.new(16384) + end + + it "cannot set invalid node sizes" do + proc { + IO::Buffer.default_node_size = 0xffffffffffffffff + }.should raise_error(RangeError) + proc { + IO::Buffer.default_node_size = 0 + }.should raise_error(ArgumentError) + proc { + IO::Buffer.new(0xffffffffffffffff) + }.should raise_error(RangeError) + proc { + IO::Buffer.new(0) + }.should raise_error(ArgumentError) + end + ####### private ####### -- Eric Wong From tony at medioh.com Sat Nov 28 12:22:22 2009 From: tony at medioh.com (Tony Arcieri) Date: Sat, 28 Nov 2009 10:22:22 -0700 Subject: [Rev-talk] [PATCH 0/2] improve 1.8 performance when mixed with threads In-Reply-To: <1259304577-6055-1-git-send-email-normalperson@yhbt.net> References: <1259304577-6055-1-git-send-email-normalperson@yhbt.net> Message-ID: This is likewise awesome. On Thu, Nov 26, 2009 at 11:49 PM, Eric Wong wrote: > Hi, > > Under 1.8, the RevThreadSpawn concurrency model of Rainbows! was performing > horribly for processing large uploads. I narrowed it down to the > blocking-for-10ms behavior of Rev under 1.8. Since the Thread running the > Rev::Loop always gets scheduled, unconditionally blocking to do nothing for > 10ms > becomes very bad. > > Now, instead of blocking for 10ms unconditionally, we now detect if we're > the > lone thread in the process and run in non-blocking mode if there are any > other > threads. This change does not affect 1.9. > > With the current rainbows.git[1], the times for the following test goes > from ~73 seconds to ~6 seconds. I'm not missing any zeros there :) > > # note(1), run "make -C t random_blob" first for more accurate timings > # note(2), I don't use Rubygems for development, so setting up RUBYLIB > # by hand to include rack, unicorn (and rev+iobuffer, of course) is > # required > make -C t models=RevThreadSpawn t0100-rack-input-hammer.sh > > The same test under 1.9 is around ~2.5 seconds. > > [1] 921a961e91276522c134813052a7b7e06b00ef2b git:// > git.bogomips.org/rainbows.git > > Both of these are pushed out to the usual location at git://yhbt.net/rev > > Eric Wong (2): > make Rev::Loop#run_nonblock signal-safe for 1.8 > Make Rev::Loop#run_once less intrusive with 1.8 threads > _______________________________________________ > Rev-talk mailing list > Rev-talk at rubyforge.org > http://rubyforge.org/mailman/listinfo/rev-talk > -- Tony Arcieri Medioh/Nagravision -------------- next part -------------- An HTML attachment was scrubbed... URL: From tony at medioh.com Sat Nov 28 12:21:55 2009 From: tony at medioh.com (Tony Arcieri) Date: Sat, 28 Nov 2009 10:21:55 -0700 Subject: [Rev-talk] [PATCH 2/2] Make Rev::Loop#run_once less intrusive with 1.8 threads In-Reply-To: <1259304577-6055-3-git-send-email-normalperson@yhbt.net> References: <1259304577-6055-1-git-send-email-normalperson@yhbt.net> <1259304577-6055-3-git-send-email-normalperson@yhbt.net> Message-ID: Nice, this is an optimization I considered doing myself but never got around to. On Thu, Nov 26, 2009 at 11:49 PM, Eric Wong wrote: > Instead of blocking for 10ms when other threads are present, > run in non-blocking mode if we're not the only thread in the > process. This change does not negatively impact 1.9. > --- > ext/rev/extconf.rb | 4 ++++ > ext/rev/rev_loop.c | 37 ++++++++++++++++++++++++++----------- > 2 files changed, 30 insertions(+), 11 deletions(-) > > diff --git a/ext/rev/extconf.rb b/ext/rev/extconf.rb > index 9b48dd9..2422cbc 100644 > --- a/ext/rev/extconf.rb > +++ b/ext/rev/extconf.rb > @@ -8,6 +8,10 @@ if have_func('rb_thread_blocking_region') > $defs << '-DHAVE_RB_THREAD_BLOCKING_REGION' > end > > +if have_func('rb_thread_alone') > + $defs << '-DHAVE_RB_THREAD_ALONE' > +end > + > if have_func('rb_str_set_len') > $defs << '-DHAVE_RB_STR_SET_LEN' > end > diff --git a/ext/rev/rev_loop.c b/ext/rev/rev_loop.c > index 86dad37..bde0021 100644 > --- a/ext/rev/rev_loop.c > +++ b/ext/rev/rev_loop.c > @@ -12,6 +12,14 @@ > > #include "rev.h" > > +#if defined(HAVE_RB_THREAD_BLOCKING_REGION) > +# define Rev_Loop_may_block_safely() (1) > +#elif defined(HAVE_RB_THREAD_ALONE) > +# define Rev_Loop_may_block_safely() (rb_thread_alone()) > +#else /* just in case Ruby changes: */ > +# define Rev_Loop_may_block_safely() (0) > +#endif > + > static VALUE mRev = Qnil; > static VALUE cRev_Loop = Qnil; > > @@ -175,19 +183,26 @@ void Rev_Loop_process_event(VALUE watcher, int > revents) > */ > static VALUE Rev_Loop_run_once(VALUE self) > { > - struct Rev_Loop *loop_data; > VALUE nevents; > - > - Data_Get_Struct(self, struct Rev_Loop, loop_data); > > - assert(loop_data->ev_loop && !loop_data->events_received); > - > - Rev_Loop_ev_loop_oneshot(loop_data); > - Rev_Loop_dispatch_events(loop_data); > - > - nevents = INT2NUM(loop_data->events_received); > - loop_data->events_received = 0; > - > + if (Rev_Loop_may_block_safely()) { > + struct Rev_Loop *loop_data; > + > + Data_Get_Struct(self, struct Rev_Loop, loop_data); > + > + assert(loop_data->ev_loop && !loop_data->events_received); > + > + Rev_Loop_ev_loop_oneshot(loop_data); > + Rev_Loop_dispatch_events(loop_data); > + > + nevents = INT2NUM(loop_data->events_received); > + loop_data->events_received = 0; > + > + } else { > + nevents = Rev_Loop_run_nonblock(self); > + rb_thread_schedule(); > + } > + > return nevents; > } > > -- > 1.6.5.3.300.g9be30 > > _______________________________________________ > Rev-talk mailing list > Rev-talk at rubyforge.org > http://rubyforge.org/mailman/listinfo/rev-talk > -- Tony Arcieri Medioh/Nagravision -------------- next part -------------- An HTML attachment was scrubbed... URL: From tony at medioh.com Sat Nov 28 12:44:41 2009 From: tony at medioh.com (Tony Arcieri) Date: Sat, 28 Nov 2009 10:44:41 -0700 Subject: [Rev-talk] [PATCH iobuffer] Allow global tuning of the default node size In-Reply-To: <20091127192840.GA16058@dcvr.yhbt.net> References: <20091127192840.GA16058@dcvr.yhbt.net> Message-ID: Merged and pushed. I bumped DEFAULT_NODE_SIZE to 16384 myself. I'll try doing a release here soon unless you anticipate making any more changes. My first release on Gemcutter, I guess. Should be interesting. On Fri, Nov 27, 2009 at 12:28 PM, Eric Wong wrote: > Hi, I've also pushed out to git://yhbt.net/iobuffer > > Background: > > Using the stupid dd(1)-like benchmark in Unicorn[1] with the Rainbows! > Rev concurrency model, I noticed Rev being significantly[2] slower > than the EventMachine one with certain response sizes, while other > (both smaller and larger, the two were roughly on par[3]. It turned > out the default benchmark defaults of writing 4096-byte response > bodies managed to cause many extra allocations because of HTTP headers > being written separately. > > Since Rev::IO::INPUT_SIZE is 16384, I figured it would make sense to > use 16384 in Rainbows!, too. This should also have the pleasant side > effect of allowing the malloc() implementation to share unused blocks > between Rev input and output more easily. I noticed this performance > improvement both using tcmalloc and glibc ptmalloc2. > > I would not hesitate changing the DEFAULT_NODE_SIZE to 16384 so > applications are less likely to want to change it, too. > > [1] git://git.bogomips.org/unicorn.git test/benchmark/dd.ru > [2] roughly half as fast as EM > [3] EM is still a little faster, further investigation needed... > > >From 715cb9915de00e857f3e727e8c49117bbf06ffb7 Mon Sep 17 00:00:00 2001 > From: Eric Wong > Date: Fri, 27 Nov 2009 11:08:09 -0800 > Subject: [PATCH] Allow global tuning of the default node size > > This is useful for applications that use Rev and do not > allocate IO::Buffer objects directly. > --- > ext/iobuffer.c | 56 > +++++++++++++++++++++++++++++++++++++++++++------- > spec/buffer_spec.rb | 25 ++++++++++++++++++++++ > 2 files changed, 73 insertions(+), 8 deletions(-) > > diff --git a/ext/iobuffer.c b/ext/iobuffer.c > index 30f798d..aa8a6ab 100644 > --- a/ext/iobuffer.c > +++ b/ext/iobuffer.c > @@ -23,6 +23,7 @@ > > /* Default number of bytes in each node's buffer. Should be >= MTU */ > #define DEFAULT_NODE_SIZE 4096 > +static unsigned default_node_size = DEFAULT_NODE_SIZE; > > struct buffer { > unsigned size, node_size; > @@ -43,6 +44,8 @@ static VALUE IO_Buffer_allocate(VALUE klass); > static void IO_Buffer_mark(struct buffer *); > static void IO_Buffer_free(struct buffer *); > > +static VALUE IO_Buffer_default_node_size(VALUE klass); > +static VALUE IO_Buffer_set_default_node_size(VALUE klass, VALUE size); > static VALUE IO_Buffer_initialize(int argc, VALUE *argv, VALUE self); > static VALUE IO_Buffer_clear(VALUE self); > static VALUE IO_Buffer_size(VALUE self); > @@ -77,6 +80,11 @@ void Init_iobuffer() > cIO_Buffer = rb_define_class_under(rb_cIO, "Buffer", rb_cObject); > rb_define_alloc_func(cIO_Buffer, IO_Buffer_allocate); > > + rb_define_singleton_method(cIO_Buffer, "default_node_size", > + IO_Buffer_default_node_size, 0); > + rb_define_singleton_method(cIO_Buffer, "default_node_size=", > + IO_Buffer_set_default_node_size, 1); > + > rb_define_method(cIO_Buffer, "initialize", IO_Buffer_initialize, -1); > rb_define_method(cIO_Buffer, "clear", IO_Buffer_clear, 0); > rb_define_method(cIO_Buffer, "size", IO_Buffer_size, 0); > @@ -108,29 +116,61 @@ static void IO_Buffer_free(struct buffer *buf) > } > > /** > + * call-seq: > + * IO_Buffer.default_node_size -> 4096 > + * > + * Retrieves the current value of the default node size. > + */ > +static VALUE IO_Buffer_default_node_size(VALUE klass) > +{ > + return UINT2NUM(default_node_size); > +} > + > +/* > + * safely converts node sizes from Ruby numerics to C and raising > + * ArgumentError or RangeError on invalid sizes > + */ > +static unsigned convert_node_size(VALUE size) > +{ > + int node_size = NUM2INT(size); > + > + if(node_size < 1) rb_raise(rb_eArgError, "invalid buffer size"); > + > + return (unsigned)node_size; > +} > + > +/** > + * call-seq: > + * IO_Buffer.default_node_size = 16384 > + * > + * Sets the default node size for calling IO::Buffer.new with no > arguments. > + */ > +static VALUE IO_Buffer_set_default_node_size(VALUE klass, VALUE size) > +{ > + default_node_size = convert_node_size(size); > + > + return size; > +} > + > +/** > * call-seq: > - * IO_Buffer.new(size = DEFAULT_NODE_SIZE) -> IO_Buffer > + * IO_Buffer.new(size = IO::Buffer.default_node_size) -> IO_Buffer > * > * Create a new IO_Buffer with linked segments of the given size > */ > static VALUE IO_Buffer_initialize(int argc, VALUE *argv, VALUE self) > { > VALUE node_size_obj; > - int node_size; > struct buffer *buf; > > if(rb_scan_args(argc, argv, "01", &node_size_obj) == 1) { > - node_size = NUM2INT(node_size_obj); > - > - if(node_size < 1) rb_raise(rb_eArgError, "invalid buffer size"); > - > Data_Get_Struct(self, struct buffer, buf); > > /* Make sure we're not changing the buffer size after data has been > allocated */ > assert(!buf->head); > assert(!buf->pool_head); > > - buf->node_size = node_size; > + buf->node_size = convert_node_size(node_size_obj); > } > > return Qnil; > @@ -329,7 +369,7 @@ static struct buffer *buffer_new(void) > buf = (struct buffer *)xmalloc(sizeof(struct buffer)); > buf->head = buf->tail = buf->pool_head = buf->pool_tail = 0; > buf->size = 0; > - buf->node_size = DEFAULT_NODE_SIZE; > + buf->node_size = default_node_size; > > return buf; > } > diff --git a/spec/buffer_spec.rb b/spec/buffer_spec.rb > index 4329bf6..fa38740 100644 > --- a/spec/buffer_spec.rb > +++ b/spec/buffer_spec.rb > @@ -98,6 +98,31 @@ describe IO::Buffer do > @buffer.should_not be_empty > end > > + it "can set default node size" do > + IO::Buffer.default_node_size = 1 > + IO::Buffer.default_node_size.should == 1 > + (IO::Buffer.default_node_size = 4096).should == 4096 > + end > + > + it "can be created with a different node size" do > + IO::Buffer.new(16384) > + end > + > + it "cannot set invalid node sizes" do > + proc { > + IO::Buffer.default_node_size = 0xffffffffffffffff > + }.should raise_error(RangeError) > + proc { > + IO::Buffer.default_node_size = 0 > + }.should raise_error(ArgumentError) > + proc { > + IO::Buffer.new(0xffffffffffffffff) > + }.should raise_error(RangeError) > + proc { > + IO::Buffer.new(0) > + }.should raise_error(ArgumentError) > + end > + > ####### > private > ####### > -- > Eric Wong > _______________________________________________ > Rev-talk mailing list > Rev-talk at rubyforge.org > http://rubyforge.org/mailman/listinfo/rev-talk > -- Tony Arcieri Medioh/Nagravision -------------- next part -------------- An HTML attachment was scrubbed... URL: From normalperson at yhbt.net Sat Nov 28 13:14:35 2009 From: normalperson at yhbt.net (Eric Wong) Date: Sat, 28 Nov 2009 18:14:35 +0000 Subject: [Rev-talk] [PATCH iobuffer] Allow global tuning of the default node size In-Reply-To: References: <20091127192840.GA16058@dcvr.yhbt.net> Message-ID: <20091128181435.GA18486@dcvr.yhbt.net> Tony Arcieri wrote: > Merged and pushed. I bumped DEFAULT_NODE_SIZE to 16384 myself. > > I'll try doing a release here soon unless you anticipate making any more > changes. My first release on Gemcutter, I guess. Should be interesting. Awesome, thanks. I don't think I have time or need to look into more performance tuning here at the moment. -- Eric Wong From tony at medioh.com Sat Nov 28 13:34:24 2009 From: tony at medioh.com (Tony Arcieri) Date: Sat, 28 Nov 2009 11:34:24 -0700 Subject: [Rev-talk] [PATCH iobuffer] Allow global tuning of the default node size In-Reply-To: <20091128181435.GA18486@dcvr.yhbt.net> References: <20091127192840.GA16058@dcvr.yhbt.net> <20091128181435.GA18486@dcvr.yhbt.net> Message-ID: Okay, while as it stands I was investigating how to push stuff to Gemcutter and already went ahead and pushed it. :) It's now released as iobuffer 0.1.2 On Sat, Nov 28, 2009 at 11:14 AM, Eric Wong wrote: > Tony Arcieri wrote: > > Merged and pushed. I bumped DEFAULT_NODE_SIZE to 16384 myself. > > > > I'll try doing a release here soon unless you anticipate making any more > > changes. My first release on Gemcutter, I guess. Should be interesting. > > Awesome, thanks. I don't think I have time or need to look into more > performance tuning here at the moment. > > -- > Eric Wong > -- Tony Arcieri Medioh/Nagravision -------------- next part -------------- An HTML attachment was scrubbed... URL: From normalperson at yhbt.net Sat Nov 28 15:31:13 2009 From: normalperson at yhbt.net (Eric Wong) Date: Sat, 28 Nov 2009 12:31:13 -0800 Subject: [Rev-talk] [PATCH iobuffer] Allow global tuning of the default node size In-Reply-To: References: <20091127192840.GA16058@dcvr.yhbt.net> <20091128181435.GA18486@dcvr.yhbt.net> Message-ID: <20091128203113.GA23201@dcvr.yhbt.net> Tony Arcieri wrote: > Okay, while as it stands I was investigating how to push stuff to Gemcutter > and already went ahead and pushed it. :) > > It's now released as iobuffer 0.1.2 That explains why iobuffer.o already exists when I unpack the gem :> "gem install" does not work out-of-the-box for me: make cc -shared -o iobuffer.so iobuffer.o -L. -L/home/ew/lib -L. -Wl,-O1 -rdynamic -Wl,-export-dynamic -lc -ldl -lcrypt -lm -lc iobuffer.o: file not recognized: File format not recognized collect2: ld returned 1 exit status make: *** [iobuffer.so] Error 1 -- Eric Wong From tony at medioh.com Sat Nov 28 16:01:06 2009 From: tony at medioh.com (Tony Arcieri) Date: Sat, 28 Nov 2009 14:01:06 -0700 Subject: [Rev-talk] [PATCH iobuffer] Allow global tuning of the default node size In-Reply-To: <20091128203113.GA23201@dcvr.yhbt.net> References: <20091127192840.GA16058@dcvr.yhbt.net> <20091128181435.GA18486@dcvr.yhbt.net> <20091128203113.GA23201@dcvr.yhbt.net> Message-ID: On Sat, Nov 28, 2009 at 1:31 PM, Eric Wong wrote: > Tony Arcieri wrote: > > Okay, while as it stands I was investigating how to push stuff to > Gemcutter > > and already went ahead and pushed it. :) > > > > It's now released as iobuffer 0.1.2 > > That explains why iobuffer.o already exists when I unpack the gem :> > > "gem install" does not work out-of-the-box for me: > > make > cc -shared -o iobuffer.so iobuffer.o -L. -L/home/ew/lib -L. -Wl,-O1 > -rdynamic -Wl,-export-dynamic -lc -ldl -lcrypt -lm -lc > iobuffer.o: file not recognized: File format not recognized > collect2: ld returned 1 exit status > make: *** [iobuffer.so] Error 1 > Oof... Yeah, the build/release system is definitely all ad hoc and nasty. Perhaps I should look at switching to hoe or echoe or whatever the new hotness is for packaging/releasing gems. -- Tony Arcieri Medioh/Nagravision -------------- next part -------------- An HTML attachment was scrubbed... URL: From tony at medioh.com Sat Nov 28 16:11:31 2009 From: tony at medioh.com (Tony Arcieri) Date: Sat, 28 Nov 2009 14:11:31 -0700 Subject: [Rev-talk] [PATCH iobuffer] Allow global tuning of the default node size In-Reply-To: References: <20091127192840.GA16058@dcvr.yhbt.net> <20091128181435.GA18486@dcvr.yhbt.net> <20091128203113.GA23201@dcvr.yhbt.net> Message-ID: Just redid the release... I updated the gemspec to only glob .c and .rb files. This should be sufficient for iobuffer and prevent future botched releases like this in the future. For rev it will be a bit more involved, and I've similarly botched such releases in the past. Anyway, just pushed iobuffer 0.1.3. Let me know if that works. On Sat, Nov 28, 2009 at 2:01 PM, Tony Arcieri wrote: > On Sat, Nov 28, 2009 at 1:31 PM, Eric Wong wrote: > >> Tony Arcieri wrote: >> > Okay, while as it stands I was investigating how to push stuff to >> Gemcutter >> > and already went ahead and pushed it. :) >> > >> > It's now released as iobuffer 0.1.2 >> >> That explains why iobuffer.o already exists when I unpack the gem :> >> >> "gem install" does not work out-of-the-box for me: >> >> make >> cc -shared -o iobuffer.so iobuffer.o -L. -L/home/ew/lib -L. -Wl,-O1 >> -rdynamic -Wl,-export-dynamic -lc -ldl -lcrypt -lm -lc >> iobuffer.o: file not recognized: File format not recognized >> collect2: ld returned 1 exit status >> make: *** [iobuffer.so] Error 1 >> > > Oof... > > Yeah, the build/release system is definitely all ad hoc and nasty. > > Perhaps I should look at switching to hoe or echoe or whatever the new > hotness is for packaging/releasing gems. > > -- > Tony Arcieri > Medioh/Nagravision > -- Tony Arcieri Medioh/Nagravision -------------- next part -------------- An HTML attachment was scrubbed... URL: From normalperson at yhbt.net Sat Nov 28 16:14:03 2009 From: normalperson at yhbt.net (Eric Wong) Date: Sat, 28 Nov 2009 21:14:03 +0000 Subject: [Rev-talk] [PATCH iobuffer] Allow global tuning of the default node size In-Reply-To: References: <20091127192840.GA16058@dcvr.yhbt.net> <20091128181435.GA18486@dcvr.yhbt.net> <20091128203113.GA23201@dcvr.yhbt.net> Message-ID: <20091128211403.GA24222@dcvr.yhbt.net> Tony Arcieri wrote: > On Sat, Nov 28, 2009 at 1:31 PM, Eric Wong wrote: > > Tony Arcieri wrote: > > > Okay, while as it stands I was investigating how to push stuff to > > Gemcutter > > > and already went ahead and pushed it. :) > > > > > > It's now released as iobuffer 0.1.2 > > > > That explains why iobuffer.o already exists when I unpack the gem :> > > > > "gem install" does not work out-of-the-box for me: > > > > make > > cc -shared -o iobuffer.so iobuffer.o -L. -L/home/ew/lib -L. -Wl,-O1 > > -rdynamic -Wl,-export-dynamic -lc -ldl -lcrypt -lm -lc > > iobuffer.o: file not recognized: File format not recognized > > collect2: ld returned 1 exit status > > make: *** [iobuffer.so] Error 1 > > > > Oof... > > Yeah, the build/release system is definitely all ad hoc and nasty. > > Perhaps I should look at switching to hoe or echoe or whatever the new > hotness is for packaging/releasing gems. I got confused/annoyed by little things in both Hoe and Echoe and just decided to go with GNU makefiles in most of my projects :> I'm not afraid to admit that I actually *like* programming in gmake, and it lets me run tests in parallel and I prefer writing Bourne shell for a lot of things, too. -- Eric Wong From normalperson at yhbt.net Sat Nov 28 16:15:30 2009 From: normalperson at yhbt.net (Eric Wong) Date: Sat, 28 Nov 2009 21:15:30 +0000 Subject: [Rev-talk] [PATCH iobuffer] Allow global tuning of the default node size In-Reply-To: References: <20091127192840.GA16058@dcvr.yhbt.net> <20091128181435.GA18486@dcvr.yhbt.net> <20091128203113.GA23201@dcvr.yhbt.net> Message-ID: <20091128211530.GB24222@dcvr.yhbt.net> Tony Arcieri wrote: > Anyway, just pushed iobuffer 0.1.3. Let me know if that works. Yup, works on both Linux x86 and x86_64 Thanks! -- Eric Wong