[zeromq-dev] ZeroMQ pattern queueing messages

Emil Stenqvist emil at unomaly.com
Wed Jan 29 16:17:40 CET 2014


Hi

I'm building a frontend service that will receive syslog messages (up to
300,000 msgs/s, or 20MB/s of data), attach a timestamp to them and
distribute them evenly to a pool of workers for processing and storage.
Once a worker has received an event it doesn't send a reply, so from the
frontend's viewpoint it's send-and-forget. (That is -- we can accept losing
a few messages in case of worker crashes.)

The service is designed so that one thread listens for syslog data, which
it then sends to another thread that deals with workers. I've chosen the
PUSH-PULL pattern for frontend-to-worker communication.

However, I also want the frontend to queue up incoming messages in case
there are no workers available, for which I'm pondering these two options:

1. Using ZeroMQ for queueing: e.g. an in-process DEALER-ROUTER socket
combination and where I adjust the HWM value to keep receiving and storing
messages while waiting for new workers.
2. Implement the queue structure myself and just use ZeroMQ for shuffling
data between threads

Option 1 seems justified here:

http://lists.zeromq.org/pipermail/zeromq-dev/2010-March/002838.html

where Martin says that they've seen in-memory queues of up to 200GB. Is
this a "good" approach? I'd like to see a) how many messages are in the
queue, and possibly b) start dropping old messages once the HWM is reached.
Is there any way to do this with ZeroMQ? Or should I handle this queueing
behavior outside of ZeroMQ?

I'd appreciate all pointers and/or hints!

Cheers
Emil
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.zeromq.org/pipermail/zeromq-dev/attachments/20140129/32ea859c/attachment.htm>


More information about the zeromq-dev mailing list