Shared memory between workers

Eric Wong normalperson at yhbt.net
Mon Apr 26 15:03:39 EDT 2010


Iñaki Baz Castillo <ibc at aliax.net> wrote:
> Hi, I plan to build a SIP TCP server (no UDP) based on
> Unicorn/Rainbows! HTTP server. The main different between a SIP server
> and HTTP server are:
> 
> - SIP uses persistent TCP connections, so I should use Rainbows!.
> - For a SIP server it's not valid a simple request-response model.
> Different workers could handle SIP messages (requests and responses)
> belonging to the same SIP session so I need a shared memory between
> all the workers.
> 
> Another option is using EventMachine, perhaps more suitable for this
> purpose by design as it uses a single Ruby process so sharing memory
> is not a problem. In the other side using a single process in a
> multicore server is a pain.
> I would like to use Unicorn/Rainbows as I love its design: by far it's
> the more reliable and efficient Ruby HTTP server and it takes
> advantages of Unix's features.
> 
> I don't want to use a DB server neither MemCache as "shared memory" as
> it would be too slow. Is there any way to share RAM memory between
> different Unicorn/Rainbows! workers in a *safe* way? I could create a
> Hash or Array of SIP sessions into the master process so all the
> workers reuse it, but I don't think it would be safe to access/write
> into it from different Ruby processes. For that I would also need a
> semaphore system (perhaps again a shared variable between all workers
> in order to lock the shared Array/Hash when a worker writes into it).
> 
> Any tip about it? suggstions?

Hi Iñaki,

First off I wouldn't call Memcached slow...  How many requests do you
need out of it and what kind of access patterns are you doing?

If you're on a modern Linux 2.6 box, then anything on the local
filesystem is basically shared memory if you have enough RAM to store
your working set (and it sounds like you do).

If you're willing to be confined to a single box, I've had great
experiences with TokyoCabinet the past few years.  One system I built
shares 10G of read-mostly data (on 16G boxes) across 8 Unicorn
(previously Mongrel) workers.  TokyoCabinet uses mmap and pread/pwrite,
so you can safely share DB handles across any number of native
threads/processes.

Writes in TC need filesystem locking and you can only have one active
writer, but that was still plenty fast enough in my experience.

If fsync() becomes a bottleneck, then for non-critical data either:

  1) disable it (TC lets you)
  2) or use libeatmydata if you choose something that doesn't let
     you disable fsync()

<soapbox>
  My personal opinion is that fsync()/O_SYNC are completely overrated.
  All important machines these days have redundant storage, redundant
  power supplies, ECC memory, hardware health monitoring/alerting and
  backup power.  Any remaining hardware/kernel failure possibilities
  are far less likely than application bugs that corrupt data :)

  But, if you really have critical data, /then/ get an SSD or
  a battery-backed write cache.
</soapbox>

-- 
Eric Wong


More information about the mongrel-unicorn mailing list