[zeromq-dev] publish/subscribe and the "fast producer" problem

Doron Somech somdoron at gmail.com
Sat Mar 4 10:31:38 CET 2017


I think it will be easy to tweak the libzmq code yourself to support this
behavior. Try to do it with a new socket option and make a pull request. I
can help you with finding the right place in the code.

On Mar 3, 2017 23:20, "Marcin Romaszewicz" <marcin at brkt.com> wrote:

> Yeah, you need a way to throttle back the publisher. Have the many nodes
> tell the publisher to slow down if they detect too high of a drop rate.
> Your publisher can then decide whether to throttle for the slowest client,
> or for the fastest client. Also, pub/sub may not be the right way to do
> this for 1:many if the value of 'many' is very high, because each message
> you send will be sent 1:1 to each subscriber, slowing them all down. Some
> kind of tree based fanout may work better.
>
> -- Marcin
>
> On Fri, Mar 3, 2017 at 12:49 PM, Justin Karneges <justin at karneges.com>
> wrote:
>
>> And what if there are no subscribers? Publishing should block?
>>
>> It sounds like your goal is some kind of sensible flow control (I'm not
>> sure what else an "at least 1 subscriber can read" use-case really is). My
>> advice is to publish at a fixed rate, for example by sleeping between
>> publishes.
>>
>> I do wish ZMQ_RATE worked for all PUB sockets and not just PGM-based.
>>
>> On Fri, Mar 3, 2017, at 12:31 PM, Francesco wrote:
>>
>> Hi all,
>>
>> I'm new to ZeroMQ but so far all my experiments with it have been quite
>> positive, so thanks for your great work!
>>
>> However, one thing that I cannot really sort out is the following: I have
>> written a small sample program where I create a ZMQ_PUB socket and I
>> continuously zmq_msg_send() messages in it, never sleeping. This is
>> simulating what I will do in a larger program where I plan to use ZeroMQ.
>> Such program is a massively-parallel utility that needs to send in a fanout
>> fashion (1 to many) several short messages per second (up to say 1million
>> messages / second).
>>
>> In my scenario I don't care about subscribers joining late or eventually
>> loosing some messages (if the network is slow or the subscriber itself is
>> too slow). I do care however about detecting such conditions where messages
>> are dropped.
>>
>> My problem is the "fast producer" one: this sample program shows that the
>> zmq_msg_send() never returns an error; doing some math such my little
>> utility says that is publishing data at rates up to 400Gbps...  the only
>> problem is that the NIC on the computer is a 1Gbps NIC. Of course all
>> subscribers report missing 99% of messages (I put a sequence number in the
>> messages I send).
>>
>> This problem does not appear to be new, some interesting references I
>> found are:
>>
>>    http://grokbase.com/t/zeromq/zeromq-dev/11ca11s9b9/pub-sub-
>> pattern-rate-control-and-backpressure            (Dec 2011)
>>    http://grokbase.com/t/zeromq/zeromq-dev/12797gy703/notify-s
>> end-er-that-theyve-hit-the-high-water-mark      (Jul 2012)
>>    http://grokbase.com/t/zeromq/zeromq-dev/129n2e2sx5/high-wat
>> er-mark-notification-for-publisher                   (Sep 2012)
>>
>> I experimented a little bit with ZeroMQ and found that:
>>   - ZMQ_RATE is not doing anything
>>   - ZMQ_XPUB_NODROP set to 1 works BUT basically ties the publisher to
>> the SLOWEST subscriber: from my experiments it looks like (despite the
>> documentation) the publisher now blocks every time there is just 1
>> subscriber queue that has hit the HWM
>>
>> as written in some of these posts what I would like to have is a NODROP
>> socket option that allows my publisher to send data as fast as the FASTEST
>> subscriber can handle.
>>
>> So here's my question: is there any way to achieve the above, i.e. block
>> the publisher if ALL subscriber queues hit their HWM ?
>>
>>
>> Thanks a lot for any hint!
>>
>> Francesco
>>
>>
>>
>>
>>
>> *_______________________________________________*
>> zeromq-dev mailing list
>> zeromq-dev at lists.zeromq.org
>> https://lists.zeromq.org/mailman/listinfo/zeromq-dev
>>
>>
>>
>> _______________________________________________
>> zeromq-dev mailing list
>> zeromq-dev at lists.zeromq.org
>> https://lists.zeromq.org/mailman/listinfo/zeromq-dev
>>
>
>
> _______________________________________________
> zeromq-dev mailing list
> zeromq-dev at lists.zeromq.org
> https://lists.zeromq.org/mailman/listinfo/zeromq-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.zeromq.org/pipermail/zeromq-dev/attachments/20170304/9bf48ba6/attachment.htm>


More information about the zeromq-dev mailing list