[zeromq-dev] publish/subscribe and the "fast producer" problem

Justin Karneges justin at karneges.com
Tue Mar 7 07:39:07 CET 2017

On Mon, Mar 6, 2017, at 01:26 AM, Francesco wrote:
> Hi Justin,
> >> > My advice
> >> > is to publish at a fixed rate, for example by sleeping between publishes.
> >>
> >> Unfortunately this is not a solution in my case, as the events I
> >> publish are generated by other incoming events that I cannot control
> >> and in general I will not know at which rate they arrive.
> >
> > Ah, but even if your writes automatically slowed down to the fastest
> > subscriber, you'd still face this problem. What will you do with
> > incoming events while you're waiting for the fastest subscriber to
> > become writable again?
> In such a case I just need to drop input events.

If you want to drop events to subscribers that can't read, including
events to the fastest subscriber, then you simply use PUB. ;)

I think this moves us into the realm of diagnostics. You want to know if
the publisher is dropping messages.

It might be handy if there were an alternative to ZMQ_XPUB_NODROP (say
ZMQ_XPUB_ATLEASTONE) that required at least one subscriber to be
writable, but not all of them. Then you could send with ZMQ_DONTWAIT,
and an EAGAIN error would indicate that not even the fastest subscriber
was writable (you could then discard the message yourself and not retry
sending it).

To solve your problem without changing ZeroMQ, you might make a
subscriber feedback system. For example, add a sequence ID to the
published payloads, and if a subscriber sees an ID out of sequence then
it can keep a running count of missed messages. Periodically, each
subscriber should submit the number of messages it has missed (or zero
if none missed) to the publisher (via a separate socket, like PUSH) and
then zero out its counter. As long as the publisher regularly receives
messages saying zero missed, then you know there's at least one
subscriber out there keeping up. If nobody is saying zero missed, then
you've gone over.

More information about the zeromq-dev mailing list