[zeromq-dev] Publish-subscribe / Time to live
Pieter Hintjens
ph at imatix.com
Thu Jul 29 17:26:24 CEST 2010
Hi Eric,
Is it correct to assume the back-end workers are stateless?
What you're describing (a job control system) will require some
development on top of 0MQ. You would not use pubsub sockets, they are
for broadcast-style data distribution.
What you would need to make, presumably, is a broker that accepts job
requests (as complete messages) from clients, using req/xrep sockets,
and then distributes these to worker processes (or threads if you want
to do it all in one process) using xreq/rep sockets. The broker would
have the necessary logic to do the preparatory tasks, and then the
analysis, put the results together, and send this back to the client.
To handle disconnecting clients I can see two options. Either ignore
them, and discard results that cannot be sent back to a client, or add
a Keep-Alive from the client to the broker, and purge jobs for a
client that goes away.
This is a very sketchy design. If you'd like us to make a detailed
design, our pleasure, but that would be a few days' work and we'd
charge for that.
Best regards
--
Pieter Hintjens
iMatix
On Thu, Jul 29, 2010 at 5:02 PM, Eric Bell <eric at ericjbell.com> wrote:
>
> I want to implement a message queueing system that has some specific features, and I'm a little confused about what ZeroMQ does and doesn't do.
>
> I am implementing a data analysis web application that uses back-end workers to perform analysis tasks. These tasks run for a second to a minute. A job consists of ten to twenty tasks, a couple of which have to be performed sequentially first, and then all the others can run concurrently as resources allow. I need to deliver status and analysis results back to the client; which is the browser.
>
> The simplest approach would be to have JobStatus and JobResults queues, into which the status and results messages are placed. The client would at some point subscribe to these queues (via a web gateway), and be able to grab any messages in the queue relevant to the client's job. I see two potential problems, and I just want to make sure I understand how ZeroMQ works.
>
> 1) With a publish-subscribe pattern, can clients subscribe and unsubscribe at any time? I assume that clients will only see messages that were queued after the time they subscribe. What needs to happen when a client unsubscribes, so messages don't build up in the queueing system waiting for a subscriber who is no longer there?
>
> 2) Is there a way to set a time-to-live on messages, so that they expire if they aren't consumed within a certain period of time?
>
> The relevance to my problem is that if my user (a person) closes their browser, then any messages that were being held for them need to be discarded. A time-out on messages would be the easiest, because I won't have to do anything special to close down ZeroMQ subscribers. Without that, I will need to construct a process that determines when the client has "gone away", so it can perform any clean-up work required by ZeroMQ.
>
> Thanks!
>
> --eric
>
>
> _______________________________________________
> zeromq-dev mailing list
> zeromq-dev at lists.zeromq.org
> http://lists.zeromq.org/mailman/listinfo/zeromq-dev
>
More information about the zeromq-dev
mailing list