Posted By: Adam Pisoni
Date: 2008-01-17 16:19
Summary: Skynet 0.9.2 nearing release
Project: Skynet Map/Reduce System
There are some exciting changes in the works for Skynet 0.9.2, due to be released in less than 2 weeks.
First of all, we're moving the subversion repository over to sourceforge and opening up development to others. I hope you consider getting involved.
Some of the major improvements include:
- Support for multiple arbitrary queues
- Streaming task generation for jobs with large data sets
- Tighter control over failover and retries.
- Enforced timeouts (dramatically improving performance)
- LOTS of bug fixes.
- New Skynet::Job single mode, where the master job executes tasks locally. Included in this are settings to control minimum thresholds where master jobs will execute map or reduce tasks locally.
- Optimizations to ActiveRecord.distributed_find for dealing with large data sets.
So keep your eyes open for a release in the next week or so. |
|