Forums | Admin Discussion Forums: help Start New Thread Nested Flat Threaded Ultimate Show 25 Show 50 Show 75 Show 100   By: Dave Grijalva RE: How are intermediate files handled in Sky [ reply ]   2008-12-16 21:09 This depends on your message_queue_adapter. Intermediate values are put back into the message queue. If you are using the mysql message queue adapter, the values are inserted into the database. When and if the content is written to disk is handled completely by mysql at that point. If you are using the tuplespace message queue adapter, you can read the source within the skynet source to find out when/if it writes to disk. I think it probably does not. Neither of these guarantees the OS doesn't write the data to the drive as swap depending on memory conditions. By: Adam Pisoni RE: How are intermediate files handled in Sky [ reply ]   2008-09-23 00:00 Well, I suppose the database may write it to disk, but Skynet does not write it to disk. By: S. M. RE: How are intermediate files handled in Sky [ reply ]   2008-09-22 21:16 Thank you, Adam. You say "at some point"... is there any guarantee that there is in fact no writing to disk between the map and reduce phases? Thanks again, -Suzanne By: Adam Pisoni RE: How are intermediate files handled in Sky [ reply ]   2008-09-22 19:15 Mappers return their values via the skynet message queue. They are then picked up by the worker that is holding the master task/job for that job and partitioned for the reduce phase. So, at some point all returned map results are held in memory so they can be partitioned. By: S. M. How are intermediate files handled in SkyNet? [ reply ]   2008-09-22 19:06 Hi, Are intermediate <key, value> pairs output by the map phase written to disk before the reduce phase? or is it stored in memory? Thanks, -SM