[zeromq-dev] ZeroMQ File Server

[Thomas S Hatch]

On Sat, May 14, 2011 at 1:59 PM, Thomas S Hatch <thatch45 at gmail.com> wrote:

> Excellent, this is the info I need, now I should be able to figure it out.
> Thanks!
>
> I will probably have more questions, I will post it once I get it figured
> out. My plan is to make a simple api to embed a simple zmq file server into
> a python app. Then use it in salt.
> On May 14, 2011 1:38 PM, "Pieter Hintjens" <ph at imatix.com> wrote:
> > On Sat, May 14, 2011 at 5:12 PM, Thomas S Hatch <thatch45 at gmail.com>
> wrote:
> >
> >> Great! This is what I was hoping to hear! If the entire file is getting
> read
> >> into ram anyway, how would I go about making a file server that would
> send
> >> the files in chunks and not read the whole thing into ram? I think that
> you
> >> can see me intention here, but I gues I am still not entirely sure how
> to do
> >> it yet.
> >
> > So here's one design. You read/write files in chunks of say 1MB, each
> > a 0MQ message. Your protocol to send a file is:
> >
> > * OPEN filename - requests to send a new file, reply gives a staging
> > handle and the size of the file already received, if GT 0,
> > * STAGE message - sends a chunk to a staging handle.
> > * DELIVER handle - closes the file staging, publishes the file
> >
> > I'd use a router/dealer socket pair so that I can do the open and
> > deliver commands synchronously, and the stage command asynchronously.
> >
> > This gives you a good level of performance and lets you send massive
> > files that don't fit in memory, with recovery if there's a network
> > issue or node crash.
> >
> > If you're sending 10K config files, of course, this is overkill, just
> > slurp and send the file in one message.
> >
> > -Pieter
> > _______________________________________________
> > zeromq-dev mailing list
> > zeromq-dev at lists.zeromq.org
> > http://lists.zeromq.org/mailman/listinfo/zeromq-dev
>

Ok, I think that this is on the right track, can you all let me know/ give
suggestions?
https://github.com/thatch45/sandbox/blob/master/examples/zmq/file_get2.py
https://github.com/thatch45/sandbox/blob/master/examples/zmq/file_serve2.py

obviously there is a ways to go, since this is still the same level of
simplicity as the last attempt, but this should at least give the chunks
seperately
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.zeromq.org/pipermail/zeromq-dev/attachments/20110519/a64b265f/attachment.htm>