From pardavimai at internetsolutions.lt  Mon Jun  4 06:18:33 2012
From: pardavimai at internetsolutions.lt (=?UTF-8?B?UGFrbGF1c2ltYXMgZMSXbCBpbnRlcm5ldG8gc3ZldGFpbsSXcyAtIEFLQ0lKQQ==?=)
Date: Mon, 04 Jun 2012 09:18:33 +0300
Subject: Sveiki..
Message-ID: <7da13ed2f0d598e7b045e4bad8c36269@naujienlaiskiai.eu>

Laba diena,

Noriu informuoti kad ?iuo metu vyksta AKCIJA - iki 2012.07.01 nuolaidos
svetaini? talpinimo planams naujiems klientams!

Kuriame interneto svetaines (nemokamai), klientai moka tik u? svetain?s
talpinim? m?s? serveryje.
Atsinaujinusius svetaini? talpinimo planus ir ?kainius rasite pateiktoje
nuorodoje:
http://internetsolutions.lt/tinklapiu-kurimas
Galime susikurti svetain? ir be talpinimo paslaugos, d?l kain?
teirautis nurodytais kontaktais.

Jei jau turite svetain?, galime pasi?lyti svetain?s optimizacij? (SEO
paslaugas).
Pla?iau apie tai galite rasti m?s? tinklapyje:
http://internetsolutions.lt/reklama-internete/seo-paslaugos/


Geros dienos,




--
Pagarbiai
Tomas Aleksiejus
UAB "Internet Solutions"
verslo klient? vadybininkas
tel.: +370 686 01033
pardavimai at internetsolutions.lt
www.internetsolutions.lt
Reg. adresas: Tvenkini? g. 8A, ?aki? k., Kauno r. sav.
Biuro - korespondencijos adresas: Lyros g. 5?57, ?iauliai


From kunalm at gmail.com  Thu Jun  7 01:30:11 2012
From: kunalm at gmail.com (Kunal Modi)
Date: Wed, 6 Jun 2012 18:30:11 -0700
Subject: Rainbows on Heroku Issues
Message-ID: <CADtMPNGx4-Xa5o6w98DGJV96=0E-VOs=nSi8xiFUoTN64Xr_YA@mail.gmail.com>

Hi,

I am trying to run Rainbows on Heroku and I'm getting some odd errors,
I was wondering if anyone had any explanation / idea on how to solve.
I created a simple Rails 3.1/Ruby 1.9.3 app to test Unicorn/Rainbows
on Heroku (just a simple endpoint that responds with simple json). The
app worked as expected with Unicorn, but has some odd tendencies on
Heroku when using Rainbows.

The request completely processes in Heroku, but always waits
keepalive_timeout time before actually returning the response:
2012-06-06T22:33:01+00:00 app[web.1]: Started GET "/" at 2012-06-06
22:33:01 +0000
2012-06-06T22:33:01+00:00 app[web.1]: cache: [GET /] miss
2012-06-06T22:33:06+00:00 heroku[router]: GET
simple-waterfall-3300.herokuapp.com/?dyno=web.1 queue=0 wait=0ms
service=5020ms status=200 bytes=13

When I changed keepalive timeout to 10s, the responses were similarly
delayed by 10s. When I set the keelalive timeout to 0, Heroku returned
2012-06-07T00:50:02+00:00 heroku[router]: Error H13 (Connection closed
without response)


My Rainbows specific config was:
? use :ThreadPool
? worker_connections 30
??keepalive_timeout 5



Thanks a lot!
Kunal

From normalperson at yhbt.net  Thu Jun  7 04:17:49 2012
From: normalperson at yhbt.net (Eric Wong)
Date: Thu, 7 Jun 2012 04:17:49 +0000
Subject: Rainbows on Heroku Issues
In-Reply-To: <CADtMPNGx4-Xa5o6w98DGJV96=0E-VOs=nSi8xiFUoTN64Xr_YA@mail.gmail.com>
References: <CADtMPNGx4-Xa5o6w98DGJV96=0E-VOs=nSi8xiFUoTN64Xr_YA@mail.gmail.com>
Message-ID: <20120607041749.GA14953@dcvr.yhbt.net>

Kunal Modi <kunalm at gmail.com> wrote:
> I am trying to run Rainbows on Heroku and I'm getting some odd errors,
> I was wondering if anyone had any explanation / idea on how to solve.
> I created a simple Rails 3.1/Ruby 1.9.3 app to test Unicorn/Rainbows
> on Heroku (just a simple endpoint that responds with simple json). The
> app worked as expected with Unicorn, but has some odd tendencies on
> Heroku when using Rainbows.
> 
> The request completely processes in Heroku, but always waits
> keepalive_timeout time before actually returning the response:

Try adding Rack::ContentLength and/or Rack::Chunked middleware.

You might be missing a Content-Length or "Transfer-Encoding: chunked"
header in the response, so the client is assuming HTTP/1.0 semantics
and waiting for the connection to close.  HTTP/1.1 requires either
Content-Length or chunked transfer-encoding to do persistent
connections.

From superjoe at gmail.com  Fri Jun  8 16:04:48 2012
From: superjoe at gmail.com (Joseph McDonald)
Date: Fri, 8 Jun 2012 09:04:48 -0700
Subject: rainbows slower?
Message-ID: <CAKN4AWvqRyxTMOVNjc5bGzrkOKGZJ6TsTwBuYu_ZekXaeqT9ag@mail.gmail.com>

Hi,

I have a "hello world" sinatra app, benchmark:
ab -n 1000 -c 100 http://$mysite/

Total transferred:      264000 bytes
HTML transferred:       13000 bytes
Requests per second:    308.76 [#/sec] (mean)
Time per request:       323.871 [ms] (mean)
Time per request:       3.239 [ms] (mean, across all concurrent requests)

If I put rainbows in front of it, I get:

Total transferred:      272000 bytes
HTML transferred:       13000 bytes
Requests per second:    61.62 [#/sec] (mean)
Time per request:       1622.745 [ms] (mean)
Time per request:       16.227 [ms] (mean, across all concurrent requests)

rainbow config looks like:

worker_processes 7  # i'm on an 8 core system
Rainbows! do
  use :EventMachine   # sinatra app using "thin"
  worker_connections 1024
end

config.ru looks like:
require './webserver'
run Sinatra::Application

ruby version is: ruby 1.9.3p194

I tried telling apachebench to use keepalives (-k) and that slowed
rainbows down more (and sped up the straight sinatra app).
I have changed worker_processes and worker_connections with no effect.
 I added keepalive_timeout 0 to the Rainbows! block with no effect.

any idea why rainbows is slower?  I'd like to take advantage of all my cores.

thanks,
Joe

From normalperson at yhbt.net  Fri Jun  8 17:39:20 2012
From: normalperson at yhbt.net (Eric Wong)
Date: Fri, 8 Jun 2012 17:39:20 +0000
Subject: rainbows slower?
In-Reply-To: <CAKN4AWvqRyxTMOVNjc5bGzrkOKGZJ6TsTwBuYu_ZekXaeqT9ag@mail.gmail.com>
References: <CAKN4AWvqRyxTMOVNjc5bGzrkOKGZJ6TsTwBuYu_ZekXaeqT9ag@mail.gmail.com>
Message-ID: <20120608173920.GA20100@dcvr.yhbt.net>

Joseph McDonald <superjoe at gmail.com> wrote:
> Hi,
> 
> I have a "hello world" sinatra app, benchmark:
> ab -n 1000 -c 100 http://$mysite/
> 
> Total transferred:      264000 bytes
> HTML transferred:       13000 bytes
> Requests per second:    308.76 [#/sec] (mean)
> Time per request:       323.871 [ms] (mean)
> Time per request:       3.239 [ms] (mean, across all concurrent requests)
> 
> If I put rainbows in front of it, I get:
> 
> Total transferred:      272000 bytes
> HTML transferred:       13000 bytes
> Requests per second:    61.62 [#/sec] (mean)
> Time per request:       1622.745 [ms] (mean)
> Time per request:       16.227 [ms] (mean, across all concurrent requests)
> 
> rainbow config looks like:
> 
> worker_processes 7  # i'm on an 8 core system
> Rainbows! do
>   use :EventMachine   # sinatra app using "thin"
>   worker_connections 1024
> end

What is your RACK_ENV set to?  I'm not sure how Thin sets it or defaults
to, but you can pass the "-E" switch to Rainbows!:

  rainbows -E none ...

(or use the RACK_ENV environment variable)

Iff you're on Linux, and RACK_ENV doesn't help, can you try:

  listen host_port, :tcp_defer_accept => false

I still need to investigate what (I think) is a bug in Linux which
makes TCP_DEFER_ACCEPT behave erratically.

> config.ru looks like:
> require './webserver'
> run Sinatra::Application

Can you show us your application?

> ruby version is: ruby 1.9.3p194
> 
> I tried telling apachebench to use keepalives (-k) and that slowed
> rainbows down more (and sped up the straight sinatra app).
> I have changed worker_processes and worker_connections with no effect.
>  I added keepalive_timeout 0 to the Rainbows! block with no effect.

Can you also verify your app is setting Content-Length (or
Transfer-Encoding: chunked) in its responses?

> any idea why rainbows is slower?  I'd like to take advantage of all my
> cores.

I've listed what I can think of above.  Thanks for giving Rainbows!
a short.  I'd like to figure out why performance is worse, too.

From normalperson at yhbt.net  Fri Jun  8 23:31:31 2012
From: normalperson at yhbt.net (Eric Wong)
Date: Fri, 8 Jun 2012 16:31:31 -0700
Subject: rainbows slower?
In-Reply-To: <20120608173920.GA20100@dcvr.yhbt.net>
References: <CAKN4AWvqRyxTMOVNjc5bGzrkOKGZJ6TsTwBuYu_ZekXaeqT9ag@mail.gmail.com>
	<20120608173920.GA20100@dcvr.yhbt.net>
Message-ID: <20120608233131.GB6086@dcvr.yhbt.net>

Eric Wong <normalperson at yhbt.net> wrote:
> What is your RACK_ENV set to?  I'm not sure how Thin sets it or defaults
> to, but you can pass the "-E" switch to Rainbows!:
> 
>   rainbows -E none ...
> 
> (or use the RACK_ENV environment variable)

I suspect it's RACK_ENV being "development" by default and slowing
things down heavily (Rack::Lint).  I wasn't able to reproduce the
drastic differences between thin and Rainbows with RACK_ENV=none on
Rack::Lobster.  Not a real app I know, but in the absence of a real one,
it's all I've got.

I used the following config.ru
------------------------------
require "rack/lobster"
use Rack::ContentLength
use Rack::ContentType
run Rack::Lobster.new
------------------------------

and my rainbows.conf.rb
------------------------------
worker_processes 2 # only 2 cores on this machine, I also tried with 1
Rainbows! do
  use :EventMachine
  worker_connections 1024
end
------------------------------

thin 1.3.1
rainbows 4.3.1
rack 1.4.1
Ruby 1.9.3-p194
Linux 3.4.1 on x86-64

Commands used:
	ab -n 1000 -c 100 -k http://127.0.0.1:3000/
	RACK_ENV=none thin start -R config.ru
	RACK_ENV=none rainbows -c rainbows.conf.rb config.ru -p 3000

I tested without keepalives (-k), too but again, it's too close
to say one server is faster than another based on Rack::Lobster.
For "hello world"-type apps, Rainbows! will use worker_processes
more effectively with keepalives.

From alexandre at objectdata.com.br  Mon Jun 18 18:32:00 2012
From: alexandre at objectdata.com.br (Alexandre Riveira)
Date: Mon, 18 Jun 2012 18:32:00 +0000
Subject: background threads
Message-ID: <4FDF7420.7030300@objectdata.com.br>

Hello !

I'm trying to put in background threads in Rainbows. I created a 
config/initializers/background_thread.rb, where I

Thread.new do
# my background job
end

With that, I have problems to "reload" in rainbows

Thank you all for your help.


Alexandre Riveira
Brazil

From normalperson at yhbt.net  Mon Jun 18 21:55:50 2012
From: normalperson at yhbt.net (Eric Wong)
Date: Mon, 18 Jun 2012 14:55:50 -0700
Subject: background threads
In-Reply-To: <4FDF7420.7030300@objectdata.com.br>
References: <4FDF7420.7030300@objectdata.com.br>
Message-ID: <20120618215550.GA31162@dcvr.yhbt.net>

Alexandre Riveira <alexandre at objectdata.com.br> wrote:
> Hello !
> 
> I'm trying to put in background threads in Rainbows. I created a
> config/initializers/background_thread.rb, where I
> 
> Thread.new do
> # my background job
> end
> 
> With that, I have problems to "reload" in rainbows

What kind of problems?

Is preload_app true or false?  (default is false)

preload_app being true will cause that background thread to get
started in the master process and not the workers.  You can
workaround it by starting th background thread in an after_fork
hook in the Rainbows!/unicorn config file:

http://unicorn.bogomips.org/Unicorn/Configurator.html#method-i-after_fork

> Thank you all for your help.

I'm often lazy when it comes to setting up hooks myself :x

So for background threads, I prefer to start it in the app stack
on the first request (use a mutex to protect against double-starts).
For rails, you can probably do it in a before_filter.

Raindrops::Watcher is a Rack app (no other framework, just rack)
which starts a background thread in the #call method:

	git clone git://bogomips.org/raindrops raindrops
	cat raindrops/lib/watcher.rb

From alexandre at objectdata.com.br  Mon Jun 18 19:08:16 2012
From: alexandre at objectdata.com.br (Alexandre Riveira)
Date: Mon, 18 Jun 2012 19:08:16 +0000
Subject: background threads
In-Reply-To: <20120618215550.GA31162@dcvr.yhbt.net>
References: <4FDF7420.7030300@objectdata.com.br>
	<20120618215550.GA31162@dcvr.yhbt.net>
Message-ID: <4FDF7CA0.6030103@objectdata.com.br>

Em 18-06-2012 21:55, Eric Wong escreveu:

What kind of problems?

The application consumes 100% cpu after reload


Is preload_app true or false?  (default is false)
preload_app is true


I changed  my code  as below  and got  a result in  small servers
  Thread.new(log) do |rotina_log|
     loop do
        log.info("rotina automatica") {"waiting process before end"}
        sleep(5)
        break unless File.exists? Dir.pwd + '/tmp/pids/rainbows.pid.oldbin'
     end

     loop do
       #rotina_log.info("rotina automatica") { "waiting 60 secs #{Process.pid}" }
       sleep(60)
       if $0.include? "old"
         rotina_log.info("rotina automatica") { "exit..... #{Process.pid}" }
         Thread.exit
       end
       begin
         #rotina_log.info("rotina automatica") { "exec #{Process.pid}" }
         empresas = Empresa::Base.find_all_by_cancelado(false)
         #sleep(180)
         empresas.invoke(:rotinas_automaticas, :usuario_id =>  Auth::Configuracao.instance.usuario_sistema_id)
         #rotina_log.info("rotina automatica") { "exec ok #{Process.pid}" }
       rescue Exception =>  e
         rotina_log.fatal("rotina automatica") { e.to_s + e.backtrace.inspect }
       end
     end
   end


Tanks,


Alexandre Riveira
Brazil


From normalperson at yhbt.net  Mon Jun 18 22:21:05 2012
From: normalperson at yhbt.net (Eric Wong)
Date: Mon, 18 Jun 2012 22:21:05 +0000
Subject: background threads
In-Reply-To: <4FDF7CA0.6030103@objectdata.com.br>
References: <4FDF7420.7030300@objectdata.com.br>
	<20120618215550.GA31162@dcvr.yhbt.net>
	<4FDF7CA0.6030103@objectdata.com.br>
Message-ID: <20120618222105.GA460@dcvr.yhbt.net>

Alexandre Riveira <alexandre at objectdata.com.br> wrote:
> Em 18-06-2012 21:55, Eric Wong escreveu:
> 
> What kind of problems?
> 
> The application consumes 100% cpu after reload
> 
> 
> Is preload_app true or false?  (default is false)
> preload_app is true

With preload_app true, the application worker won't be able to
communicate with the background thread after reload.

I'm not sure if your setup relies on threads communicating
directly with the application.

> I changed  my code  as below  and got  a result in  small servers
>  Thread.new(log) do |rotina_log|
>     loop do
>        log.info("rotina automatica") {"waiting process before end"}
>        sleep(5)
>        break unless File.exists? Dir.pwd + '/tmp/pids/rainbows.pid.oldbin'
>     end
> 
>     loop do
>       #rotina_log.info("rotina automatica") { "waiting 60 secs #{Process.pid}" }
>       sleep(60)
>       if $0.include? "old"
>         rotina_log.info("rotina automatica") { "exit..... #{Process.pid}" }
>         Thread.exit
>       end
>       begin
>         #rotina_log.info("rotina automatica") { "exec #{Process.pid}" }
>         empresas = Empresa::Base.find_all_by_cancelado(false)
>         #sleep(180)
>         empresas.invoke(:rotinas_automaticas, :usuario_id =>  Auth::Configuracao.instance.usuario_sistema_id)
>         #rotina_log.info("rotina automatica") { "exec ok #{Process.pid}" }
>       rescue Exception =>  e
>         rotina_log.fatal("rotina automatica") { e.to_s + e.backtrace.inspect }
>       end
>     end
>   end

Reading the code above (but not knowing your original code), I'm not
seeing any communcation between the threads.  It could probably be done
better as a standalone daemon or even a cronjob.  There's more setup
overhead, sure, but it should also be easier to debug/troubleshoot.

From alexandre at objectdata.com.br  Mon Jun 18 19:26:57 2012
From: alexandre at objectdata.com.br (Alexandre Riveira)
Date: Mon, 18 Jun 2012 19:26:57 +0000
Subject: background threads
In-Reply-To: <20120618215550.GA31162@dcvr.yhbt.net>
References: <4FDF7420.7030300@objectdata.com.br>
	<20120618215550.GA31162@dcvr.yhbt.net>
Message-ID: <4FDF8101.9020500@objectdata.com.br>

Em 18-06-2012 21:55, Eric Wong escreveu:
> So for background threads, I prefer to start it in the app stack
> on the first request (use a mutex to protect against double-starts).
> For rails, you can probably do it in a before_filter.
Eric, como voc? garante a aplica??o n?o criar uma thread em cada worker

Tanks

Alexandre Riveira
brazil

From normalperson at yhbt.net  Mon Jun 18 22:42:45 2012
From: normalperson at yhbt.net (Eric Wong)
Date: Mon, 18 Jun 2012 22:42:45 +0000
Subject: background threads
In-Reply-To: <4FDF81A5.6080605@objectdata.com.br>
References: <4FDF7420.7030300@objectdata.com.br>
	<20120618215550.GA31162@dcvr.yhbt.net>
	<4FDF81A5.6080605@objectdata.com.br>
Message-ID: <20120618224245.GA1830@dcvr.yhbt.net>

Alexandre Riveira <alexandre at objectdata.com.br> wrote:
> Em 18-06-2012 21:55, Eric Wong escreveu:
> >So for background threads, I prefer to start it in the app stack
> >on the first request (use a mutex to protect against double-starts).
> >For rails, you can probably do it in a before_filter.
> 
> Eric, how do you ensure the application does not create a thread for
> each worker

In the Raindrops::Watcher case, I don't.  Watcher works as a single
process and doesn't need/care about multiple background threads.

If you only want one background thread for all workers, it probably is
best to just use a separate daemon independent of the Rainbows! server.

From normalperson at yhbt.net  Mon Jun 18 22:53:24 2012
From: normalperson at yhbt.net (Eric Wong)
Date: Mon, 18 Jun 2012 22:53:24 +0000
Subject: background threads
In-Reply-To: <4FDF85C2.1030900@objectdata.com.br>
References: <4FDF7420.7030300@objectdata.com.br>
	<20120618215550.GA31162@dcvr.yhbt.net>
	<4FDF81A5.6080605@objectdata.com.br>
	<20120618224245.GA1830@dcvr.yhbt.net>
	<4FDF85C2.1030900@objectdata.com.br>
Message-ID: <20120618225324.GD3151@dcvr.yhbt.net>

Alexandre Riveira <alexandre at objectdata.com.br> wrote:
> Em 18-06-2012 22:42, Eric Wong escreveu:
> >If you only want one background thread for all workers, it
> >probably is best to just use a separate daemon independent of the
> >Rainbows! server.
> I want only one worker to run a background thread. I would like to
> ensure that only one worker thread will be in the background. Sure
> ... using rainbows :)

It's a bad idea, I think...

You can use process-shared locks Ruby provides File#flock.
It's also possible to get fcntl()-based locking to work with pack().

Maybe there's extensions to share locks with multiple processes
(e.g. POSIX semaphores, database-based locks in Redis/MySQL/Postgres).

...Or even use a really ugly method of using TCPServer to bind to a
pre-determined port (and bailing on Errno::EADDRINUSE) (no need to
accept connections on the TCPServer, just keep the object around
so it doesn't get close(2)-ed on GC.

But really, I think this type of background thread is a bad idea
for Rainbows! (or any HTTP server).

From ivmaykov at gmail.com  Tue Jun 19 09:06:19 2012
From: ivmaykov at gmail.com (Ilya Maykov)
Date: Tue, 19 Jun 2012 02:06:19 -0700
Subject: Rainbows! + EventMachine + Sinatra::Synchrony == pegged CPU when idle?
Message-ID: <CAAsVqN8=HxMbbQkfonojsoCHgJAPsDhC7hqqcBK9GnYsO2ww7g@mail.gmail.com>

Hi all,

We're using Rainbows + EventMachine + Sinatra::Synchrony to run a
fleet of RESTful web servers backed by a Cassandra cluster. We are
using the EventMachineTransport to talk to Cassandra with an
EM::Synchrony::ConnectionPool in each rainbows worker. We have a Storm
cluster pushing a large stream of real-time data into the Rainbows
fleet using HTTP PUT requests. We're running into some very strange
performance issues and need help figuring out what's going on.

Basically, when load is low, everything looks good. When we crank up
the load, all of a sudden the CPU gets pegged, request latencies go
waaaay up, and requests start timing out. Once this state is reached,
the high CPU usage (4 rainbows worker processes at ~50% each on a
2-core machine = nearly full load) remains even if we completely shut
off all incoming traffic. Taking a look with strace -p, it looks like
the rainbows processes are writing ascii NUL characters to file
descriptor 7 (which is a FIFO) as fast as the kernel will let them. My
guess is that the worker is trying to communicate with the rainbows
master process via the FIFO. Not sure what is triggering this
behavior, but would like to know if anyone else has ever seen
something like this. This thread sounded like it could've been a
similar issue, but died out without any conclusion:
http://rubyforge.org/pipermail/rainbows-talk/2012-April/000345.html

Some details about the setup:

6-node cassandra cluster
3 nodes running rainbows web servers
4 rainbows workers per node
max of 50 cassandra connections per rainbows worker
rainbows.conf has:

Rainbows! do
  use :EventMachine
  worker_connections 50
  keepalive_requests 1000
  keepalive_timeout  10
end

So, each rainbows node can handle 4 * 50 = 200 simultaneous connections

12 Storm worker processes writing to the rainbows web servers
each Storm worker has max of 10 connections open to each of the 3 rainbows nodes

So, each rainbows node has 12 * 10 = 120 incoming connections from Storm.

Have been playing around with the numbers, the bug (assuming it is a
bug) seems to be easier to trigger when I increase the number of
incoming connections (from Storm workers), even if they are a lot less
than the rainbows servers can take (60-70% of the max connections is
usually enough). The bug is also easier to trigger when we increase
the volume of data we're pushing through Storm - hundreds or thousands
of requests per minute, no bug - hundreds of thousands of requests per
minute, yes bug. Cassandra is not the issue, it can easily take the
write load we're generating and is basically idle.

Any help in figuring this out would be greatly appreciated. Thanks,

-- Ilya

From normalperson at yhbt.net  Tue Jun 19 17:54:09 2012
From: normalperson at yhbt.net (Eric Wong)
Date: Tue, 19 Jun 2012 17:54:09 +0000
Subject: Rainbows! + EventMachine + Sinatra::Synchrony == pegged CPU
	when idle?
In-Reply-To: <CAAsVqN8=HxMbbQkfonojsoCHgJAPsDhC7hqqcBK9GnYsO2ww7g@mail.gmail.com>
References: <CAAsVqN8=HxMbbQkfonojsoCHgJAPsDhC7hqqcBK9GnYsO2ww7g@mail.gmail.com>
Message-ID: <20120619175409.GA27303@dcvr.yhbt.net>

Ilya Maykov <ivmaykov at gmail.com> wrote:
> Hi all,
> 
> We're using Rainbows + EventMachine + Sinatra::Synchrony to run a
> fleet of RESTful web servers backed by a Cassandra cluster. We are
> using the EventMachineTransport to talk to Cassandra with an
> EM::Synchrony::ConnectionPool in each rainbows worker. We have a Storm
> cluster pushing a large stream of real-time data into the Rainbows
> fleet using HTTP PUT requests. We're running into some very strange
> performance issues and need help figuring out what's going on.

I'm not at all familiar with Storm nor Cassandra.  How big are the HTTP
PUT requests Rainbows! is getting?  Is Storm pipelining HTTP requests by
any chance?  That may not do well with the EM portion of Rainbows!

> Basically, when load is low, everything looks good. When we crank up
> the load, all of a sudden the CPU gets pegged, request latencies go
> waaaay up, and requests start timing out. Once this state is reached,
> the high CPU usage (4 rainbows worker processes at ~50% each on a
> 2-core machine = nearly full load) remains even if we completely shut
> off all incoming traffic.

> Taking a look with strace -p, it looks like
> the rainbows processes are writing ascii NUL characters to file
> descriptor 7 (which is a FIFO) as fast as the kernel will let them. My
> guess is that the worker is trying to communicate with the rainbows
> master process via the FIFO.

No, rainbows doesn't have code to send "\0" to the master.  I
don't think EM does, either, maybe some other library you're
using...

Which version of Ruby is this?  Try adding "-f" to follow threads
for a worker.

> Not sure what is triggering this
> behavior, but would like to know if anyone else has ever seen
> something like this. This thread sounded like it could've been a
> similar issue, but died out without any conclusion:
> http://rubyforge.org/pipermail/rainbows-talk/2012-April/000345.html
> 
> Some details about the setup:
> 
> 6-node cassandra cluster
> 3 nodes running rainbows web servers
> 4 rainbows workers per node
> max of 50 cassandra connections per rainbows worker
> rainbows.conf has:
> 
> Rainbows! do
>   use :EventMachine
>   worker_connections 50
>   keepalive_requests 1000
>   keepalive_timeout  10
> end

Do you have preload_app set to true anywhere?  (Try leaving it as false
(the default))

Can you also try "keepalive_timeout 0" to disable keepalive?  (EM
handles it internally, but I'm not sure how well)

> So, each rainbows node can handle 4 * 50 = 200 simultaneous connections
> 
> 12 Storm worker processes writing to the rainbows web servers
> each Storm worker has max of 10 connections open to each of the 3 rainbows nodes
> 
> So, each rainbows node has 12 * 10 = 120 incoming connections from Storm.
> 
> Have been playing around with the numbers, the bug (assuming it is a
> bug) seems to be easier to trigger when I increase the number of
> incoming connections (from Storm workers), even if they are a lot less
> than the rainbows servers can take (60-70% of the max connections is
> usually enough). The bug is also easier to trigger when we increase
> the volume of data we're pushing through Storm - hundreds or thousands
> of requests per minute, no bug - hundreds of thousands of requests per
> minute, yes bug. Cassandra is not the issue, it can easily take the
> write load we're generating and is basically idle.
> 
> Any help in figuring this out would be greatly appreciated. Thanks,

Try my suggestions above.

I would also search your libs/gems for what's writing "\0" since I don't
think it's Rainbows!...

From masterkain at gmail.com  Fri Jun 22 17:55:49 2012
From: masterkain at gmail.com (Claudio Poli)
Date: Fri, 22 Jun 2012 19:55:49 +0200
Subject: undefined method `each' for nil:NilClass (NoMethodError)
Message-ID: <CANp6Qo+CO6mF=+CrEKcPRRxCP=b335Bw=YCYOnAGSFeD2CV_Gg@mail.gmail.com>

Hello,
I started getting this exception a lot: http://pastie.org/4133611

Probably happens when a client interrupts the connection.

Any clue?

Thanks.

From normalperson at yhbt.net  Fri Jun 22 19:34:46 2012
From: normalperson at yhbt.net (Eric Wong)
Date: Fri, 22 Jun 2012 19:34:46 +0000
Subject: undefined method `each' for nil:NilClass (NoMethodError)
In-Reply-To: <CANp6Qo+CO6mF=+CrEKcPRRxCP=b335Bw=YCYOnAGSFeD2CV_Gg@mail.gmail.com>
References: <CANp6Qo+CO6mF=+CrEKcPRRxCP=b335Bw=YCYOnAGSFeD2CV_Gg@mail.gmail.com>
Message-ID: <20120622193446.GA20228@dcvr.yhbt.net>

Claudio Poli <masterkain at gmail.com> wrote:
> Hello,
> I started getting this exception a lot: http://pastie.org/4133611
> 
> Probably happens when a client interrupts the connection.
> 
> Any clue?

Hi Claudio,

What can you tell us about your application?  What framework,
environment, versions are you using?

>From what I see, something in your Rack application stack is causing
an invalid Rack response (nil in the body).   A valid Rack response
is a 3 part array:

0 - status
    #to_i makes it an HTTP status code (e.g. 200, "200 OK")

1 - headers
    #each yields through a pair of stringifiable objects
    (e.g. Rack::Utils::HeaderHash, Hash)

2 - body (this is where you're getting a nil, which is wrong)
    This must respond to #each and yield a single string
    (e.g. Array, IO...)
    May optionally respond to :to_path and :closes

-- 
Eric Wong