[zeromq-dev] Multiplexing on top of TCP and subports

[Martin Sustrik]

Hi Ian,

Sorry for the delay. I'm a bit overloaded with work these days.

>         if there's some value in hit HWM generating an upstream pub
>         message on XPUB sockets. If you could signal the subscription
>         set that
>         the connection used, it would be easy to build systems to
>         automatically
>         spin up more downstream consumers
>
>
>     That doesn't make sense in case of PUB/SUB IMO. Messages are not
>     load-balanced, rather passed to all peers, meaning that having more
>     consumers doesn't help in handling the load in any way, it may even
>     make it worse.
>
>
> That's true throughout a tree, but almost always a PUB/SUB is going to
> terminate in leafs of the tree that are actually *doing* something with
> the messages - if those nodes can't do whatever it is fast enough then
> they're going to consumer more slowly, which causes backpressure. At the
> moment this is handled by having limited HWM and dropping extra
> messages, this was just a general thought about having a smaller HWM,
> with the ability to make improve the rate the queues drain at the leaves.
>
>         or throttle message production
>
>
>     The problem here is that this way a single slow/dead consumer can
>     completely throttle the publisher and thus stop the flow of messages
>     even to all the well behaved consumers.
>
>
> True again, but I think there are a bunch of use cases where you could
> throttle effectively without locking up the entire tree. For example,
> sensor measurements for a variety of different sensors, you could sample
> less frequently on specific ones if your processing network is unable to
> keep up. The whole tree is only necessarily blocked if you have
> homogeneous data, which is probably more common, but not necessarily
> true. I kind of think the problem you mention in the SP post regarding
> buffer bloat is what I am referring to - if you want mostly reliable
> delivery under normal conditions but with variable load, you end up over
> allocating HWMs. If instead you could signal rates through the system,
> you could tune publishing levels to where you get better latency by
> sending somewhat less data.

Right.

The both cases boil down to ensuring that the message throughput in all 
parts of the tree is the same.

For example: "Throttle the producer so that it doesn't exceed the agreed 
upon message rate."

Or: "Assign more resources to the consumer node so that it can keep up 
with the agreed upon message rate."


There are two subproblems to this AFAICS:

1. How do we communicate the "agreed upon message rate" throughout the 
distribution tree?

2. How do we notify the applications that they are not adhering the 
message rate contract?

Martin