[zeromq-dev] publish/subscribe and the "fast producer" problem

Francesco francesco.montorsi at gmail.com
Sun Mar 5 13:03:18 CET 2017


Hi Justin,
below my replies:

2017-03-03 21:49 GMT+01:00 Justin Karneges <justin at karneges.com>:
> And what if there are no subscribers? Publishing should block?
IMO in such case the publisher is free to publish messages at any rate
(in my case, 400Gbps!!)... after all... who cares if there are no
subscribers? :)


> It sounds like your goal is some kind of sensible flow control (I'm not sure
> what else an "at least 1 subscriber can read" use-case really is).
Yes exactly: the problem is that the only type of "flow control" that
can be done now (ZMQ_XPUB_NODROP=1) leads to a scenario that does not
scale well: if I have 1k subscribers and suddenly just one of them
slows down to 1message/sec for whatever reason (networking issue,
software bug, whatever), the publisher would slow down to 1message/sec
basically propagating the failure of 1 subscriber to the whole
network.

Of course also the opposite case exists: if I have 1k subscribers and
suddenly just one of them increases its processing speed by a 10x
factor, the publisher would follow it publishing 10x more messages per
sec potentially causing other subscribers to drop lots of messages.
However I think this case is much more unlikely than the first one.


> My advice
> is to publish at a fixed rate, for example by sleeping between publishes.

Unfortunately this is not a solution in my case, as the events I
publish are generated by other incoming events that I cannot control
and in general I will not know at which rate they arrive.

Thanks,
Francesco



More information about the zeromq-dev mailing list