[zeromq-dev] publish/subscribe and the "fast producer" problem

[Francesco]

Hi Marcin,

2017-03-03 22:20 GMT+01:00 Marcin Romaszewicz <marcin at brkt.com>:
> Yeah, you need a way to throttle back the publisher. Have the many nodes
> tell the publisher to slow down if they detect too high of a drop rate. Your
> publisher can then decide whether to throttle for the slowest client, or for
> the fastest client.
Yes, I think this is a possible solution... the trouble is that it
requires setting up a separate channel (something like the "reliable
pub-sub pattern" in ZeroMQ book).
My problem is also that the publisher has limited possibility of slowing down.

Anyway, regardless of which solution is adopted in case of message
drops, I think that ZeroMQ is missing an API to detect where the
message drop happens; that is I cannot know if the drop happens on the
publisher side (because the TX HWM has been hit), on the network or on
the subscriber (because its RX HWM has been hit)...
Having this information could be useful to help debugging a human to
debug the issue (while the publisher simply slow downs).

am I right saying that there is no possibility to know where the
message drop is happening?
IOW is there an API to known from ZeroMQ if some queue has dropped due
to reaching RX/TX HWM?


>Also, pub/sub may not be the right way to do this for
> 1:many if the value of 'many' is very high, because each message you send
> will be sent 1:1 to each subscriber, slowing them all down. Some kind of
> tree based fanout may work better.

yeah, in my case the networking will be 1:many, with many being in
order of 10-100 subscribers. Hopefully pub/sub will still be doable.

Thanks,
Francesco