[zeromq-dev] Slow consumer/subscriber, only get last message.

[Amr Ali]

If I were you I would put together a FIFO queue in a thread, receive early and
often, put in memory, use a 0MQ inproc socket to another thread to _pull_ from
that FIFO queue and process. 0MQ queuing should be used as a last resort, simply
because you don't want to hog your infrastructure communication (that should be
the case with most solutions).

So I'd advise that you free 0MQ from the burden early and often, however, if you
don't care that much about delivery of every message (obviously), you could
simply drop messages if 0MQ is filled up, but you also want to only show the
last message (which is unpredictable to tell which is the last message), so you
can't know which message is the last message, that brings us to the first
solution, which is free 0MQ early and often by receiving all and processing later.

Hope that helps.

On 05/10/2011 04:12 AM, Beau Trepp wrote:
> Howdy.
> 
> I'm using a PUB/SUB model to share information between a few processes (using
> 0MQ of course). However some of the consumers are much slower than others. In
> fact one is slower than the minimum publisher speed, which causes it to queue-up
> lots of messages. I can't speed up the consumer at all, so I was hoping to have
> it drop messages when it all becomes too much for it. It looks like by default
> this will happen when I have hit the high water mark. However I get the
> impression that this drops any new messages.
> 
> What I would like to happen is to drop all the messages in the consumer but the
> last one. As this consumer only needs to show information to humans, it only
> really needs the last message published. Of course to make it more difficult
> this consumer is attached to multiple publishers. So what I want is for it to
> keep the last message of each "filter" type for me to read, and the ones before
> that I don't care about.
> 
> Any ideas?. I could introduce another thread to just read 0MQ messages, but it
> seems like a waste.
> 
> Thanks,
> Beau
> 
> 
> 
> _______________________________________________
> zeromq-dev mailing list
> zeromq-dev at lists.zeromq.org
> http://lists.zeromq.org/mailman/listinfo/zeromq-dev

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 198 bytes
Desc: OpenPGP digital signature
URL: <https://lists.zeromq.org/pipermail/zeromq-dev/attachments/20110510/9919bb3d/attachment.sig>