[zeromq-dev] ZeroMQ pattern queueing messages

[Andrew Hume]

my experience is: i start with 1 and always switch to 2.
the main reason is with 1, you never know how big the queue is
and eventually this will screw you.

On Jan 29, 2014, at 7:17 AM, Emil Stenqvist <emil at unomaly.com> wrote:

> Hi
> 
> I'm building a frontend service that will receive syslog messages (up to 300,000 msgs/s, or 20MB/s of data), attach a timestamp to them and distribute them evenly to a pool of workers for processing and storage. Once a worker has received an event it doesn't send a reply, so from the frontend's viewpoint it's send-and-forget. (That is — we can accept losing a few messages in case of worker crashes.)
> 
> The service is designed so that one thread listens for syslog data, which it then sends to another thread that deals with workers. I've chosen the PUSH-PULL pattern for frontend-to-worker communication.
> 
> However, I also want the frontend to queue up incoming messages in case there are no workers available, for which I'm pondering these two options:
> 
> 1. Using ZeroMQ for queueing: e.g. an in-process DEALER-ROUTER socket combination and where I adjust the HWM value to keep receiving and storing messages while waiting for new workers.
> 2. Implement the queue structure myself and just use ZeroMQ for shuffling data between threads
> 
> Option 1 seems justified here:
> 
> http://lists.zeromq.org/pipermail/zeromq-dev/2010-March/002838.html
> 
> where Martin says that they've seen in-memory queues of up to 200GB. Is this a "good" approach? I'd like to see a) how many messages are in the queue, and possibly b) start dropping old messages once the HWM is reached. Is there any way to do this with ZeroMQ? Or should I handle this queueing behavior outside of ZeroMQ?
> 
> I'd appreciate all pointers and/or hints!
> 
> Cheers
> Emil
> _______________________________________________
> zeromq-dev mailing list
> zeromq-dev at lists.zeromq.org
> http://lists.zeromq.org/mailman/listinfo/zeromq-dev


-----------------------
Andrew Hume
949-707-1964 (VO and best)
732-420-0907 (NJ)
andrew at research.att.com



-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.zeromq.org/pipermail/zeromq-dev/attachments/20140129/e0d15e56/attachment.htm>