[zeromq-dev] publish/subscribe and the "fast producer" problem

[Andriy Drozdyuk]

If you block the pub when the sub hits their HWM, then it wouldn't be
pub-sub anymore.
The one positive property of pub-sub is that pub side never needs to worry
about the subs. This allows you to scale... to a lot of subs.
If you start placing at-least-once requirements on this, I suggest you
don't use pub-sub, but instead design your own protocol.
For example, all the queues like rabbitmq do this.
I think this is what you actually want - because sounds like you're not
willing to loose messages.
At some point you *have* to decide to shed some load, and I find this
article very nice:
http://ferd.ca/queues-don-t-fix-overload.html



On Sun, 5 Mar 2017 at 07:03 Francesco <francesco.montorsi at gmail.com> wrote:

> Hi Justin,
> below my replies:
>
> 2017-03-03 21:49 GMT+01:00 Justin Karneges <justin at karneges.com>:
> > And what if there are no subscribers? Publishing should block?
> IMO in such case the publisher is free to publish messages at any rate
> (in my case, 400Gbps!!)... after all... who cares if there are no
> subscribers? :)
>
>
> > It sounds like your goal is some kind of sensible flow control (I'm not
> sure
> > what else an "at least 1 subscriber can read" use-case really is).
> Yes exactly: the problem is that the only type of "flow control" that
> can be done now (ZMQ_XPUB_NODROP=1) leads to a scenario that does not
> scale well: if I have 1k subscribers and suddenly just one of them
> slows down to 1message/sec for whatever reason (networking issue,
> software bug, whatever), the publisher would slow down to 1message/sec
> basically propagating the failure of 1 subscriber to the whole
> network.
>
> Of course also the opposite case exists: if I have 1k subscribers and
> suddenly just one of them increases its processing speed by a 10x
> factor, the publisher would follow it publishing 10x more messages per
> sec potentially causing other subscribers to drop lots of messages.
> However I think this case is much more unlikely than the first one.
>
>
> > My advice
> > is to publish at a fixed rate, for example by sleeping between publishes.
>
> Unfortunately this is not a solution in my case, as the events I
> publish are generated by other incoming events that I cannot control
> and in general I will not know at which rate they arrive.
>
> Thanks,
> Francesco
> _______________________________________________
> zeromq-dev mailing list
> zeromq-dev at lists.zeromq.org
> https://lists.zeromq.org/mailman/listinfo/zeromq-dev
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.zeromq.org/pipermail/zeromq-dev/attachments/20170305/c0c168f1/attachment.htm>