[zeromq-dev] publish/subscribe and the "fast producer" problem

[Justin Karneges]

On Sun, Mar 5, 2017, at 04:03 AM, Francesco wrote:
> 2017-03-03 21:49 GMT+01:00 Justin Karneges <justin at karneges.com>:
> > It sounds like your goal is some kind of sensible flow control (I'm not sure
> > what else an "at least 1 subscriber can read" use-case really is).
> Yes exactly: the problem is that the only type of "flow control" that
> can be done now (ZMQ_XPUB_NODROP=1) leads to a scenario that does not
> scale well: if I have 1k subscribers and suddenly just one of them
> slows down to 1message/sec for whatever reason (networking issue,
> software bug, whatever), the publisher would slow down to 1message/sec
> basically propagating the failure of 1 subscriber to the whole
> network.
> 
> Of course also the opposite case exists: if I have 1k subscribers and
> suddenly just one of them increases its processing speed by a 10x
> factor, the publisher would follow it publishing 10x more messages per
> sec potentially causing other subscribers to drop lots of messages.
> However I think this case is much more unlikely than the first one.

Thanks, this helps confirm your issue is about flow control and not
reliability.

> > My advice
> > is to publish at a fixed rate, for example by sleeping between publishes.
> 
> Unfortunately this is not a solution in my case, as the events I
> publish are generated by other incoming events that I cannot control
> and in general I will not know at which rate they arrive.

Ah, but even if your writes automatically slowed down to the fastest
subscriber, you'd still face this problem. What will you do with
incoming events while you're waiting for the fastest subscriber to
become writable again?

Justin