[zeromq-dev] ZeroMQ File Server
ph at imatix.com
Sat May 14 21:37:59 CEST 2011
On Sat, May 14, 2011 at 5:12 PM, Thomas S Hatch <thatch45 at gmail.com> wrote:
> Great! This is what I was hoping to hear! If the entire file is getting read
> into ram anyway, how would I go about making a file server that would send
> the files in chunks and not read the whole thing into ram? I think that you
> can see me intention here, but I gues I am still not entirely sure how to do
> it yet.
So here's one design. You read/write files in chunks of say 1MB, each
a 0MQ message. Your protocol to send a file is:
* OPEN filename - requests to send a new file, reply gives a staging
handle and the size of the file already received, if GT 0,
* STAGE message - sends a chunk to a staging handle.
* DELIVER handle - closes the file staging, publishes the file
I'd use a router/dealer socket pair so that I can do the open and
deliver commands synchronously, and the stage command asynchronously.
This gives you a good level of performance and lets you send massive
files that don't fit in memory, with recovery if there's a network
issue or node crash.
If you're sending 10K config files, of course, this is overkill, just
slurp and send the file in one message.
More information about the zeromq-dev