[zeromq-dev] ZeroMQ File Server

[Pieter Hintjens]

On Sat, May 14, 2011 at 5:12 PM, Thomas S Hatch <thatch45 at gmail.com> wrote:

> Great! This is what I was hoping to hear! If the entire file is getting read
> into ram anyway, how would I go about making a file server that would send
> the files in chunks and not read the whole thing into ram? I think that you
> can see me intention here, but I gues I am still not entirely sure how to do
> it yet.

So here's one design. You read/write files in chunks of say 1MB, each
a 0MQ message. Your protocol to send a file is:

* OPEN filename - requests to send a new file, reply gives a staging
handle and the size of the file already received, if GT 0,
* STAGE message - sends a chunk to a staging handle.
* DELIVER handle - closes the file staging, publishes the file

I'd use a router/dealer socket pair so that I can do the open and
deliver commands synchronously, and the stage command asynchronously.

This gives you a good level of performance and lets you send massive
files that don't fit in memory, with recovery if there's a network
issue or node crash.

If you're sending 10K config files, of course, this is overkill, just
slurp and send the file in one message.

-Pieter