 |
Forums |
Admin  Start New Thread
By: Assaf Arkin
 RE: Too many open files [ reply ]
2008-07-27 22:22
|
Thanks.
Berkley DB would be a good choice. The store is really simple in what it needs: an index to access each message, way to create/get/delete it, and make sure you don't have to processes messing with the same message at once.
I think it will be easier to ask these questions by email.
|
By: Raels Koder
RE: Too many open files [ reply ]
2008-07-27 02:32
|
Thanks for the quick response!
In spite of your self deprecations, I think that you did some great work here and would like to extend it.
I have been making some experimental mods to improve long term storage, ssh transport and other stuff. I was thinking of looking at a Berk DB implementation for the message store... to save me some work, do you know if there are any pathological issues with that approach?
I am a little confused about how the disk store master index works as well. I am slowly slogging through it. Would it be OK to ask some questions on that? If you are willing, should I start a new thread, or should we take it off-forum?
Thanks,
-r
|
By: Assaf Arkin
RE: Too many open files [ reply ]
2008-07-27 01:52
|
This implementation was only designed for simple use cases, with the expectation that you won't be handling a lot of messages concurrently, or storing them for a significant length of time.
It simply wasn't written to scale well, and has some other points at which it breaks.
With that in mind, it turns out opening/closing files is expensive, for small messages, can take as much effort as reading/writing the message content. So given that it wasn't able to handle a lot of queued messages to begin with, I added the little optimization of at least making it faster by caching open file handles.
|
By: Raels Koder
RE: Too many open files [ reply ]
2008-07-26 18:53
|
Following up on my own earlier message...
Can you tell me why the files need to be kept open? They are closed upon delete and reused... since the MessageStore::Disk::load method checks to see if it is open anyway, couldn't the code that opens the file in the MessageStore::Disk::update method simply close the file of an older message and open the new one once MAX_OPEN_FILES has been reached?
-r
|
By: Raels Koder
RE: Too many open files [ reply ]
2008-07-26 14:20
|
I bumped into this limitation also. For various reasons, using a real database on the machine is not feasible for my application.
I noticed in the code that there is a limit on the number of files that will be open. RM seems to open and keep open many more than that. Is there an open-file "leak" or am I missing something about how it is managed?
I am quite interested in creating an adapter for Berk DB and variants, so please feel free to go into detail. It is also possible that SQLite3 might be used, but I felt I should really understand the disk based part before proceeding with any substantial changes.
Any help appreciated!
Thanks,
-r
|
By: Assaf Arkin
RE: Too many open files [ reply ]
2007-03-18 21:20
|
|
I don't recommend using the disk store if you have a lot of messages in the queue at any one time. It doesn't scale as well. I use the MySQL store myself.
|
By: Mushfeq Khan
Too many open files [ reply ]
2007-03-16 19:43
|
Using ReliableMsg-1.1.0 and the file-based store, you can induce this bug on Windows and Linux:
irb(main):001:0> require 'reliable-msg'
=> true
irb(main):002:0> q = ReliableMsg::Queue.new 'testqueue'
=> #<ReliableMsg::Queue:0x3668ca0 @queue="testqueue">
irb(main):003:0> 1000.times do |i| q.put(i) end
Errno::EMFILE: Too many open files - C:/queues/ece6bb11-b62b-0129-5745-00188b293
90c.msg
from (druby://localhost:6438) c:/ruby/lib/ruby/gems/1.8/gems/reliable-ms
g-1.1.0/lib/reliable-msg/message-store.rb:312:in `initialize'
from (druby://localhost:6438) c:/ruby/lib/ruby/gems/1.8/gems/reliable-ms
g-1.1.0/lib/reliable-msg/message-store.rb:312:in `open'
from (druby://localhost:6438) c:/ruby/lib/ruby/gems/1.8/gems/reliable-ms
g-1.1.0/lib/reliable-msg/message-store.rb:312:in `update'
from (druby://localhost:6438) c:/ruby/lib/ruby/gems/1.8/gems/reliable-ms
g-1.1.0/lib/reliable-msg/message-store.rb:300:in `each'
from (druby://localhost:6438) c:/ruby/lib/ruby/gems/1.8/gems/reliable-ms
g-1.1.0/lib/reliable-msg/message-store.rb:300:in `update'
from (druby://localhost:6438) c:/ruby/lib/ruby/gems/1.8/gems/reliable-ms
g-1.1.0/lib/reliable-msg/message-store.rb:94:in `transaction'
from (druby://localhost:6438) c:/ruby/lib/ruby/gems/1.8/gems/reliable-ms
g-1.1.0/lib/reliable-msg/queue-manager.rb:350:in `queue'
from c:/ruby/lib/ruby/gems/1.8/gems/reliable-msg-1.1.0/lib/reliable-msg/
queue.rb:147:in `put'
from c:/ruby/lib/ruby/gems/1.8/gems/reliable-msg-1.1.0/lib/reliable-msg/
client.rb:128:in `call'
from c:/ruby/lib/ruby/gems/1.8/gems/reliable-msg-1.1.0/lib/reliable-msg/
client.rb:128:in `repeated'
from c:/ruby/lib/ruby/gems/1.8/gems/reliable-msg-1.1.0/lib/reliable-msg/
queue.rb:147:in `put'
from (irb):3
from (irb):3:in `times'
from (irb):3
irb(main):004:0>
It seems to be due to the way in which files are kept open inside the message store until the data is consumed. The threshold for this seems to be remarkably low - on Windows it seems to be fewer than 500 messages at a time.
This is the queues.cfg:
---
store:
type: disk
path: C:/queues
drb:
port: 6438
acl: allow 127.0.0.1
(C: is the local hard drive)
Mushfeq.
|
|
|
