[zeromq-dev] publish/subscribe and the "fast producer" problem

Justin Karneges justin at karneges.com
Fri Mar 3 21:49:58 CET 2017


And what if there are no subscribers? Publishing should block?



It sounds like your goal is some kind of sensible flow control (I'm not
sure what else an "at least 1 subscriber can read" use-case really is).
My advice is to publish at a fixed rate, for example by sleeping between
publishes.


I do wish ZMQ_RATE worked for all PUB sockets and not just PGM-based.



On Fri, Mar 3, 2017, at 12:31 PM, Francesco wrote:

> Hi all,

> 

> I'm new to ZeroMQ but so far all my experiments with it have been
> quite positive, so thanks for your great work!
> 

> However, one thing that I cannot really sort out is the following: I
> have written a small sample program where I create a ZMQ_PUB socket
> and I continuously zmq_msg_send() messages in it, never sleeping. This
> is simulating what I will do in a larger program where I plan to use
> ZeroMQ. Such program is a massively-parallel utility that needs to
> send in a fanout fashion (1 to many) several short messages per second
> (up to say 1million messages / second).
> 

> In my scenario I don't care about subscribers joining late or
> eventually loosing some messages (if the network is slow or the
> subscriber itself is too slow). I do care however about detecting such
> conditions where messages are dropped.
> 

> My problem is the "fast producer" one: this sample program shows that
> the zmq_msg_send() never returns an error; doing some math such my
> little utility says that is publishing data at rates up to 400Gbps...
> the only problem is that the NIC on the computer is a 1Gbps NIC. Of
> course all subscribers report missing 99% of messages (I put a
> sequence number in the messages I send).
> 

> This problem does not appear to be new, some interesting references I
> found are:
> 

>    http://grokbase.com/t/zeromq/zeromq-dev/11ca11s9b9/pub-sub-pattern-rate-control-and-backpressure
>    (Dec 2011)
>    http://grokbase.com/t/zeromq/zeromq-dev/12797gy703/notify-send-er-that-theyve-hit-the-high-water-mark
>    (Jul 2012)
>    http://grokbase.com/t/zeromq/zeromq-dev/129n2e2sx5/high-water-mark-notification-for-publisher
>    (Sep 2012)
> 

> I experimented a little bit with ZeroMQ and found that:

>   - ZMQ_RATE is not doing anything

>   - ZMQ_XPUB_NODROP set to 1 works BUT basically ties the publisher to
>     the SLOWEST subscriber: from my experiments it looks like (despite
>     the documentation) the publisher now blocks every time there is
>     just 1 subscriber queue that has hit the HWM
> 

> as written in some of these posts what I would like to have is a
> NODROP socket option that allows my publisher to send data as fast as
> the FASTEST subscriber can handle.
> 

> So here's my question: is there any way to achieve the above, i.e.
> block the publisher if ALL subscriber queues hit their HWM ?
> 

> 

> Thanks a lot for any hint!

> 

> Francesco

> 

>  

> 

> 

> 

> _________________________________________________

> zeromq-dev mailing list

> zeromq-dev at lists.zeromq.org

> https://lists.zeromq.org/mailman/listinfo/zeromq-dev


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.zeromq.org/pipermail/zeromq-dev/attachments/20170303/5de271a7/attachment.htm>


More information about the zeromq-dev mailing list