Files | Admin

Notes:

Release Name: 0.5.0

Notes:
raindrops 0.5.0 - more Linux extras!

Raindrops is a real-time stats toolkit to show statistics for Rack HTTP
servers.  It is designed for preforking servers such as Rainbows! and
Unicorn, but should support any Rack HTTP server under Ruby 1.9, 1.8 and
Rubinius on platforms supporting POSIX shared memory.  It may also be
used as a generic scoreboard for sharing atomic counters across multiple
processes.

* http://raindrops.bogomips.org/
* raindrops@librelist.com
* git://bogomips.org/raindrops.git

Changes:

Portable changes:
* Raindrops are now resizable within the limits of system page size
* Raindrops::Middleware proxies +to_path+ in response bodies
* More documentation

Linux-only changes:
* Raindrops::LastDataRecv[1] Rack application
* Raindrops::Watcher[2] Rack application
* Raindrops::TCP_Info[3] class for capturing TCP connection stats
* IPv6 support for inet_diag
* faster inet_diag stats for multiple sockets

There is also a demo server running the Watcher and Middleware
components.  It's capped to 30 concurrent users, so go easy on
it:

  Raindrops::Watcher: http://raindrops-demo.bogomips.org/
  Raindrops::Middleware: http://raindrops-demo.bogomips.org/_raindrops

[1] http://raindrops.bogomips.org/Raindrops/LastDataRecv.html
[2] http://raindrops.bogomips.org/Raindrops/Watcher.html
[3] http://raindrops.bogomips.org/Raindrops/TCP_Info.html



Changes: .document | 3 +- .gitignore | 4 + .wrongdoc.yml | 4 + GIT-VERSION-GEN | 2 +- GNUmakefile | 198 +--------- Gemfile | 7 + LICENSE | 2 +- README | 64 +-- Rakefile | 104 ----- examples/linux-listener-stats.rb | 123 ++++++ examples/linux-tcp-listener-stats.rb | 44 -- examples/{config.ru => middleware.ru} | 2 +- examples/watcher.ru | 4 + examples/watcher_demo.ru | 13 + examples/zbatery.conf.rb | 13 + ext/raindrops/extconf.rb | 5 + ext/raindrops/linux_inet_diag.c | 600 +++++++++++++++++++------- ext/raindrops/linux_tcp_info.c | 170 ++++++++ ext/raindrops/my_fileno.h | 36 ++ ext/raindrops/raindrops.c | 252 ++++++++++- lib/raindrops.rb | 27 +- lib/raindrops/aggregate.rb | 8 + lib/raindrops/aggregate/last_data_recv.rb | 86 ++++ lib/raindrops/aggregate/pmq.rb | 239 ++++++++++ lib/raindrops/last_data_recv.rb | 100 +++++ lib/raindrops/linux.rb | 42 ++- lib/raindrops/middleware.rb | 153 +++++-- lib/raindrops/middleware/proxy.rb | 34 ++ lib/raindrops/struct.rb | 15 + lib/raindrops/watcher.rb | 362 ++++++++++++++++ pkg.mk | 171 ++++++++ raindrops.gemspec | 30 +- test/ipv6_enabled.rb | 10 + test/rack_unicorn.rb | 12 + test/test_aggregate_pmq.rb | 65 +++ test/test_inet_diag_socket.rb | 13 + test/test_last_data_recv_unicorn.rb | 69 +++ test/test_linux.rb | 112 +++--- test/test_linux_all_tcp_listen_stats.rb | 66 +++ test/test_linux_all_tcp_listen_stats_leak.rb | 43 ++ test/test_linux_ipv6.rb | 158 +++++++ test/test_linux_tcp_info.rb | 61 +++ test/test_middleware.rb | 17 +- test/test_middleware_unicorn.rb | 37 ++ test/test_middleware_unicorn_ipv6.rb | 37 ++ test/test_raindrops.rb | 66 +++- test/test_raindrops_gc.rb | 24 +- test/test_watcher.rb | 85 ++++ 48 files changed, 3081 insertions(+), 711 deletions(-) commit 744c4c11c58f40c925c0d36eefb2287095a2f127 Author: Eric Wong Date: Wed Mar 16 19:53:59 2011 -0700 raindrops 0.5.0 - more Linux extras! Portable changes: * Raindrops are now resizable within the limits of system page size * Raindrops::Middleware proxies +to_path+ in response bodies * More documentation Linux-only changes: * Raindrops::LastDataRecv[1] Rack application * Raindrops::Watcher[2] Rack application * Raindrops::TCP_Info[3] class for capturing TCP connection stats * IPv6 support for inet_diag * faster inet_diag stats for multiple sockets There is also a demo server running the Watcher and Middleware components. It's capped to 30 concurrent users, so go easy on it: Raindrops::Watcher: http://raindrops-demo.bogomips.org/ Raindrops::Middleware: http://raindrops-demo.bogomips.org/_raindrops [1] http://raindrops.bogomips.org/Raindrops/LastDataRecv.html [2] http://raindrops.bogomips.org/Raindrops/Watcher.html [3] http://raindrops.bogomips.org/Raindrops/TCP_Info.html commit f1b45e2f44f45122313abc88e139d49c0e39c7b7 Author: Eric Wong Date: Wed Mar 16 20:12:22 2011 -0700 watcher: retry on empty stats Those will cause Aggregate to raise Errno::EDOM commit 3c6aee0e33c53d0f270b1ceb40efe6ed29fa9232 Author: Eric Wong Date: Wed Mar 16 20:10:12 2011 -0700 test_raindrops_gc: disable by default It takes too long commit 7e3b385eb433a5b23da5742ca69804f34837f947 Author: Eric Wong Date: Wed Mar 16 20:09:38 2011 -0700 test_linux_all_tcp_listen_stats_leak: fix for 1.8 Apparently 1.8 Test::Unit doesn't like empty test classes commit a3cf985b82f5b252434c89d7e0ec6277ac8fe596 Author: Eric Wong Date: Thu Mar 17 02:47:47 2011 +0000 rdoc: document the demo URLs commit 0ab20cd7e10a3c9d4ddcbe2bde28542c350b4b0e Author: Eric Wong Date: Thu Mar 17 02:33:29 2011 +0000 watcher_demo: add more listeners to the mix git is on 9418 and my cgit UNIX domain socket commit 601fdd86fdbf847e47cdbb981afc60dafa407bd5 Author: Eric Wong Date: Thu Mar 17 02:33:28 2011 +0000 inet_diag: properly deal with INADDR_ANY binds Oops :x This was totally broken with the all-listener filter. commit ca49963f55447794c3e2254fe9dc633c9268e9be Author: Eric Wong Date: Thu Mar 17 02:33:27 2011 +0000 linux-listener-stats: all means all (UNIX and TCP listeners) No more splitting all TCP or all UNIX listeners commit 39cab8b630d41d8d002f5d4f65280bc63706c74f Author: Eric Wong Date: Thu Mar 17 02:33:26 2011 +0000 update examples for http://raindrops-demo.bogomips.org/ Whee! commit 1f29269eabd806f1e0cb9ca7fd4231d8588d7669 Author: Eric Wong Date: Thu Mar 17 02:33:25 2011 +0000 watcher: properly stream responses for <= HTTP/1.0 nginx makes HTTP/1.0 requests and expects HTTP/1.1 responses commit 57c9f70fb3c14ed94f4fb2445a8a4168f8c5253b Author: Eric Wong Date: Wed Mar 16 16:38:46 2011 -0700 update examples for Rack/Zbatery commit 3656a61bfb24ac76bc47d314327c01f9fd10a717 Author: Eric Wong Date: Wed Mar 16 16:03:56 2011 -0700 doc: fix documentation for Raindrops::InetDiagSocket class commit db9162575b885add7c3b7ab06f9c03a2ebc44a1f Author: Eric Wong Date: Wed Mar 16 15:22:42 2011 -0700 add Watcher Rack application It does streaming! commit b0be66c37bb7080b0f06ab76b0e7d3404e2f9059 Author: Eric Wong Date: Mon Mar 14 14:58:40 2011 -0700 linux-listener-stats: use timerfd if available It is more accurate than sleep and doesn't require calculating times commit 65dc45b7a13c92b27bc3589e37a3b52c29b7fbee Author: Eric Wong Date: Mon Mar 14 16:52:19 2011 -0700 linux: avoid Array#first/Array#last More confusing for me, actually... commit a9bfa55f9a1d1c96d73367d3bb46f7b9e81fbbf9 Author: Eric Wong Date: Mon Mar 14 16:38:58 2011 -0700 linux: unix_listener_stats may scan all paths This matches behavior of the TCP version. commit 724f82bb5feaf1df40b02b8f353e94496f060647 Author: Eric Wong Date: Mon Mar 14 14:50:43 2011 -0700 linux-listener-stats: allow -a for all TCP sockets Sometimes we want more of them commit 65f46737ad5641841a5e5a89f4651d160dacffc0 Author: Eric Wong Date: Sat Mar 12 12:56:16 2011 -0800 inet_diag: switch to inet_pton() for translation getaddrinfo() needs to get a list of available interfaces from the kernel with every single call (since ipv6 could've been modprobed), so it's a waste of syscalls. commit c478549ea7490de2432ed31ffee37a2bfc1d24f3 Author: Eric Wong Date: Sat Mar 12 12:56:15 2011 -0800 allow reusing netlink socket for inet_diag No need to waste resources on creating/destroying a socket. commit a4b9c8334e2bc0698f0650d275be0935e86dc13e Author: Eric Wong Date: Sat Mar 12 10:17:00 2011 +0000 inet_diag: fix signedness warnings on 32-bit I thought my compiler would be smarter :< commit 44bb459d7af7efad26540c9844b86f59088d3e13 Author: Eric Wong Date: Sat Mar 12 10:13:27 2011 +0000 inet_diag: no need to specify family The way we dump, we dump it all, it seems. commit f9f10fb9815e11d5935edc9427eb1bc05dce751f Author: Eric Wong Date: Sat Mar 12 10:04:45 2011 +0000 inet_diag: fix up IPv6 address stringification This means we can read multiple addresses at once, even IPv6 ones. commit 54e0335c80a062c2fb4ec1351975500cf778f7eb Author: Eric Wong Date: Sat Mar 12 09:43:21 2011 +0000 tests: remove unused_port function It's not needed since we don't care to rebind sockets commit 79be320c3e4baf1af4b36fb269f08448c5400e47 Author: Eric Wong Date: Sat Mar 12 09:39:49 2011 +0000 inet_diag: fall back on IPv6 lookups It's slow, but at least it works. commit bf0a3c731ec3573160b9ee40cbfd7e2817d91080 Author: Eric Wong Date: Sat Mar 12 09:26:30 2011 +0000 inet_diag: fold all_tcp_listener_stats into tcp_listener_stats No reason to have an extra method. This also speeds up the multi-listener case for tcp_listener_stats since it avoids expensive sendmsg() syscalls. commit 562e2e243e69f128c47ceb59c5b1710fa35c7081 Author: Eric Wong Date: Sat Mar 12 08:58:35 2011 +0000 inet_diag: small reorganization commit 8f12e060ce2a47f92b47d197f8daf7c094277bf3 Author: Eric Wong Date: Sat Mar 12 08:51:11 2011 +0000 inet_diag: do not set unblocking function netlink is fast and predictable in response times, so permitting interrupts would just complicate things and lead to errors. commit 3a2fe1f1330899c42deb5e714db1d5799e50a292 Author: Eric Wong Date: Sat Mar 12 08:46:07 2011 +0000 move st_table cleanup Fewer places to check for errors, we think. commit e6b5e940fdb8e2ded70a80132af6c280db02e8d8 Author: Eric Wong Date: Sat Mar 12 08:30:40 2011 +0000 inet_diag: stricter IPv6 address parsing Be stricter about invalid inputs. commit d4faac5480f6416cf92301745a9a9572bc865061 Author: Eric Wong Date: Fri Mar 11 20:56:49 2011 -0800 linux: method for dumping all TCP listener stats This is a work-in-progress and will probably be modified before the next release. commit fe2615590d5e8ac8e735200696ec8396fb3cd219 Author: Eric Wong Date: Fri Mar 11 19:45:54 2011 -0800 inet_diag: force the use of 32-bit counters 64-bit counters are unnecessarily large for tracking active or queued connections until we have IP_ROFLSCALE support :> commit f6cdffc3f8ba6dc276cff38c11d9f92aea1313ea Author: Eric Wong Date: Fri Mar 11 19:44:24 2011 -0800 inet_diag: fixup braindamage from refactoring Oops :x commit af73ca855100a9aadad9bf60de45df62e8aa44e2 Author: Eric Wong Date: Fri Mar 11 16:12:17 2011 -0800 cleanup struct initialization to avoid ifdefs Too hard to maintain. commit fac0211221d70b94c0301f977e8a6ba429d79ef8 Author: Eric Wong Date: Fri Mar 11 16:05:12 2011 -0800 use unsigned values for all counters We can't have negative values commit da18a484d7e8ae15de68298557aa63cc431a7c63 Author: Eric Wong Date: Fri Mar 11 15:58:33 2011 -0800 inet_diag: cleanup unnecessarily large struct We don't care for this address. commit cf4acabf3e8993fe59840aaa6717d65f2ca95fc9 Author: Eric Wong Date: Fri Mar 11 15:04:39 2011 -0800 inet_diag: fixup broken assertion Oops, strings are always true :x commit 70041dfa82015119b33d34b46ecbb0a5c45587cf Author: Eric Wong Date: Fri Mar 11 14:56:19 2011 -0800 inet_diag: factor out prep code We're going to experiment with something... commit 55c9127ede4114f65b0bc01cdaa1152ef21ca95a Author: Eric Wong Date: Fri Mar 11 13:49:08 2011 -0800 doc: more rdoc updates for Linux users Yes we love Linux more than other systems :> commit 57fe99f294e0fe948593beeae7cf879efe4cdd52 Author: Eric Wong Date: Fri Mar 11 13:37:48 2011 -0800 the core Raindrops class is portable, yes it is OK, not to non-Unix but I'll never care :> commit 154842a48c0baba66daf744f3be06fd8701b9638 Author: Eric Wong Date: Fri Mar 11 13:35:47 2011 -0800 tcp_info: more documentation on last_data_recv It's useful, yes. commit 666cb9c38fd132ec8bee0b7860b6cf30db0f9b41 Author: Eric Wong Date: Thu Mar 10 19:18:23 2011 -0800 .document: remove non-existent reference raindrops_ext.c never existed :o commit 035bcb9fa5ab9c95bd2f5b0515f74d8d51b87805 Author: Eric Wong Date: Thu Mar 10 18:29:10 2011 -0800 tests: only do the GC test on MRI "Advanced" GCs are typically less aggressive and typically allow much more memory to be used. commit 42e8c96aa4e4d2cf3ba82751fcc0487e7e8a0225 Author: Eric Wong Date: Thu Mar 10 18:28:18 2011 -0800 fix stupid typo in Raindrops#size= Oops, it could give the GC problems. commit e87875ba261fa5b0cd328bcf03e666e2d9078114 Author: Eric Wong Date: Thu Mar 10 17:24:02 2011 -0800 tests: ensure munmap() in children don't muck up the parent Raindrops is designed to work with forking servers after all. commit e516e45640206fa3e7864938da74a7cb5ca31715 Author: Eric Wong Date: Fri Mar 11 02:12:37 2011 +0000 support for Raindrops#size= and Raindrops#evaporate! This allows limited resizing of the Raindrops memory area since we always over-allocate due to the required page aligment for mmap. It would be nice if mremap() worked with MAP_SHARED, but it does not and triggers a bus error when attempting to access the new area. ref: https://bugzilla.kernel.org/show_bug.cgi?id=8691 commit 61962b27a51031965cef70451d369b115868fb11 Author: Eric Wong Date: Thu Mar 10 10:51:38 2011 +0000 rdoc: 100% documentation coverage! Of course, RDoc doesn't know quantity vs quality :) commit 8392f8186cd21f9190474bd6b5ac6ec58c7af96a Author: Eric Wong Date: Thu Mar 10 06:11:10 2011 +0000 test_middleware_unicorn: GC is aggressive! Oops, don't let GC close our listener before Unicorn can inherit it. commit b353e42e6f12f26a926bbb3202c8bb8e7a916d1c Author: Eric Wong Date: Wed Mar 9 21:12:32 2011 -0800 experiment with Bundler for dev dependencies It seems nice commit 485d3fa598a119b409734e0a4193d78fe50f4ff7 Author: Eric Wong Date: Wed Mar 9 17:03:20 2011 -0800 disable aggregate/pmq stuff under 1.8 POSIX message queues needs native threads to function. commit fa64d77ac096612d6bc731407066df8aa1ff79a7 Author: Eric Wong Date: Wed Mar 9 16:54:46 2011 -0800 aggregate/pmq: we need a Mutex to protect fcntl() locks fcntl() locks are per-process, so we also need something to protect individual threads within a process from stepping over each other. commit b57881c86de4e7406ca537b5285f27d7130b0073 Author: Eric Wong Date: Tue Mar 8 16:52:43 2011 -0800 linux-listener-stats: do not load aggregate for single snapshot This is only useful when looped inside screen or something similar... commit d5cbd604badd58ebd99a5e19953d20d7672da4dc Author: Eric Wong Date: Tue Mar 8 16:45:01 2011 -0800 linux-listener-stats: add optional Aggregate gem support USR1 dumps histograms, and USR2 resets the counters commit 93bb8af761dbaa8e19c4660c520db1ee7ba1c70b Author: Eric Wong Date: Tue Mar 8 16:30:47 2011 -0800 linux-listener-stats: favor narrower display if possible No need to make eyes drift :) commit 90726e5187a9053c6eb7caf90ebec1d38d4372ea Author: Eric Wong Date: Tue Mar 8 14:18:11 2011 -0800 preliminary Rack app to track last_data_recv Seems to basically work commit 96c8be2ea8830e2eb3a9108f501df52c21b42546 Author: Eric Wong Date: Fri Mar 4 17:08:13 2011 -0800 test_linux_tcp_info: add test for last_data_recv commit 4372cf8ef8203c93632cdaf163a1c923075e7d0f Author: Eric Wong Date: Fri Mar 4 16:00:38 2011 -0800 Aggregate support via POSIX message queues commit 55efe20df7f2cf594786e16e890382208a114f65 Author: Eric Wong Date: Tue Mar 1 16:36:44 2011 -0800 tests: cleaner skipping for missing components commit 26558f816ecbe25955f549355aa57aef5959a22e Author: Eric Wong Date: Tue Mar 1 16:30:24 2011 -0800 gemspec: add Rack and Unicorn as dev dependencies They are useful commit 16201f3c16b66beb9694eff2bed2f4b8ccd1f010 Author: Eric Wong Date: Tue Mar 1 14:47:42 2011 -0800 test_linux_tcp_info: workaround for ancient kernels UGH... commit 943fbb44f7b2194311ac614c3a6c54420c8dda94 Author: Eric Wong Date: Tue Mar 1 14:47:41 2011 -0800 use struct tcp_info from linux/tcp.h It's more complete for people on ancient systems where "struct tcp_info" is defined in netinet/tcp and missing tcp_ircv_rtt, tcpi_rcv_space and tcpi_total_retrans. commit d93e6b2903f2d702b5dc85de86eb91eb7794cc0d Author: Eric Wong Date: Tue Mar 1 14:07:59 2011 -0800 linux-listener-stats: add timestamps and thresholds Matches my common usage patterns commit 03c75f547db2d40ea29b5753359bb0df210834f9 Author: Eric Wong Date: Mon Feb 28 17:14:04 2011 -0800 linux-tcp-listener-stats: fix fractional second delays Sometimes sleeping for one second between reads is too much... commit 1c3e0a2575c0644a7fca02bbe865e2ce421f5adb Author: Eric Wong Date: Tue Mar 1 00:57:35 2011 +0000 linux-listener-stats: not just for TCP... This can now be used to monitor UNIX domain socket queues, too. commit 6d12a518f69f950d814eb223fe5bf1ad448dcc34 Author: Eric Wong Date: Tue Mar 1 00:50:40 2011 +0000 linux-tcp-listener-stats: synchronize stdout/stderr We want any pipe readers to see this immediately commit b080ac88303f31332252aae4b430af5445114b99 Author: Eric Wong Date: Mon Feb 28 16:29:18 2011 -0800 doc: mention tcp_diag/inet_diag for old kernels People actually need to load modules manually on older kernels :< commit 012d2fe51afd46fdddbf62f7ebba5102c7f3361a Author: Eric Wong Date: Sat Feb 26 21:02:57 2011 +0000 middleware: split out proxy class It's easier to find this way. commit ac6b83ea1c277cd798e2f627fa5eee605efab4fe Author: Eric Wong Date: Sat Feb 26 20:59:00 2011 +0000 middleware: avoid double decrement when reading stats Oops! commit e4ab2cc40bdbd1698f4bcf138e83c4823d118f81 Author: Eric Wong Date: Fri Feb 25 19:43:06 2011 -0800 proxy +to_path+ calls in the response body We need to do this for apps that depend on things like the sendfile() optimizations in Rainbows! commit fcf99bd7e3323ea99c5bfc8a3a15fbbc18cc8285 Author: Eric Wong Date: Tue Feb 22 19:24:56 2011 -0800 LICENSE: fix wording, we bundle a copy of the LGPL No need for people to download glibc to get the LGPL :> commit df3d3a0b09f05249aff9f25f558f988bfcad8874 Author: Eric Wong Date: Tue Feb 22 19:24:55 2011 -0800 add Raindrops::MAX constant This is the highest number a counter may be incremented to before it overflows. commit db192979f096d0153ad14e530e1e2e193289c7e0 Author: Eric Wong Date: Tue Feb 22 19:24:54 2011 -0800 linux: add support for TCP_INFO reporting This returns a Raindrops::TCP_Info object that wraps a tcp_info struct. commit ebc2093847705c382b4d83ed5120e44b9afad3c0 Author: Eric Wong Date: Tue Feb 22 19:24:53 2011 -0800 split out RSTRUCT* compat macros We might reuse that for other code... commit a73fe2f033a766a05ecef01bef015e2d1993bddd Author: Eric Wong Date: Tue Feb 22 19:24:52 2011 -0800 use autoload for Linux module We'll be doing more Linux-only stuff commit e10e520d47fa96cf549c7d544d6575baa8ed748a Author: Eric Wong Date: Wed Feb 16 01:31:41 2011 +0000 middleware: add ipv6 address detection for Unicorn Since Unicorn and Rainbows! support IPv6 now, it makes sense to support the rfc2732-style addresses it returns. commit e5faa4da78af196ee5abbccf197671fd8e77adad Author: Eric Wong Date: Wed Feb 16 01:31:40 2011 +0000 linux: add ipv6 support for inet_diag inet_diag already supports AF_INET6. commit 5183832d1e70cd89aab1cb8bb2e2795f0ad247c7 Author: Eric Wong Date: Tue Feb 15 15:02:04 2011 -0800 README: shorten git URLs bogomips.org is on a URL diet commit 05458d4067c1642fe89f164ce3e6466af1458043 Author: Eric Wong Date: Tue Feb 15 14:57:18 2011 -0800 test_linux: fix STRESS=1 tests commit f9233c20d453475df22ce7e4d0d098837f0d859a Author: Eric Wong Date: Tue Feb 15 10:37:06 2011 -0800 middleware: switch to ivars These are slightly faster than regular method dispatch commit 7f186515f214347bf5e146ce8d16a38f32f9cdfb Author: Eric Wong Date: Tue Feb 15 10:14:22 2011 -0800 middleware: avoid capturing block variable It's a needless allocation commit 489d08233b2ec43353e25f857e3639751f4f529b Author: Eric Wong Date: Tue Feb 15 10:13:15 2011 -0800 unindent modules No need to clutter/confuse namespace lookups commit 0344ffd5fa6c4693f5a8351e451c6a0852de81ed Author: Eric Wong Date: Tue Feb 15 09:58:09 2011 -0800 update packaging to use wrongdoc + pkg.mk No more JavaScript! commit 29216b7c3337d545e49a5ce978659d47a5f3e8b0 Author: Bryan McLellan Date: Fri Feb 11 16:46:43 2011 -0800 addrs are always an Array of Strings Acked-by: Eric Wong commit 304287f2e12e351fb5bf18e28ab3ad62a2f913e5 Author: Eric Wong Date: Sun Sep 26 06:51:57 2010 +0000 README: update for Rubinius support It works!