[zeromq-dev] publish/subscribe and the "fast producer" problem

[Francesco]

Hi all,

I'm new to ZeroMQ but so far all my experiments with it have been quite
positive, so thanks for your great work!

However, one thing that I cannot really sort out is the following: I have
written a small sample program where I create a ZMQ_PUB socket and I
continuously zmq_msg_send() messages in it, never sleeping. This is
simulating what I will do in a larger program where I plan to use ZeroMQ.
Such program is a massively-parallel utility that needs to send in a fanout
fashion (1 to many) several short messages per second (up to say 1million
messages / second).

In my scenario I don't care about subscribers joining late or eventually
loosing some messages (if the network is slow or the subscriber itself is
too slow). I do care however about detecting such conditions where messages
are dropped.

My problem is the "fast producer" one: this sample program shows that the
zmq_msg_send() never returns an error; doing some math such my little
utility says that is publishing data at rates up to 400Gbps...  the only
problem is that the NIC on the computer is a 1Gbps NIC. Of course all
subscribers report missing 99% of messages (I put a sequence number in the
messages I send).

This problem does not appear to be new, some interesting references I found
are:


http://grokbase.com/t/zeromq/zeromq-dev/11ca11s9b9/pub-sub-pattern-rate-control-and-backpressure
           (Dec 2011)

http://grokbase.com/t/zeromq/zeromq-dev/12797gy703/notify-send-er-that-theyve-hit-the-high-water-mark
     (Jul 2012)

http://grokbase.com/t/zeromq/zeromq-dev/129n2e2sx5/high-water-mark-notification-for-publisher
                  (Sep 2012)

I experimented a little bit with ZeroMQ and found that:
  - ZMQ_RATE is not doing anything
  - ZMQ_XPUB_NODROP set to 1 works BUT basically ties the publisher to the
SLOWEST subscriber: from my experiments it looks like (despite the
documentation) the publisher now blocks every time there is just 1
subscriber queue that has hit the HWM

as written in some of these posts what I would like to have is a NODROP
socket option that allows my publisher to send data as fast as the FASTEST
subscriber can handle.

So here's my question: is there any way to achieve the above, i.e. block
the publisher if ALL subscriber queues hit their HWM ?


Thanks a lot for any hint!

Francesco
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.zeromq.org/pipermail/zeromq-dev/attachments/20170303/c803c16e/attachment.htm>