[zeromq-dev] Multiplexing on top of TCP and subports

[Ian Barber]

On Wed, Sep 28, 2011 at 4:40 PM, Martin Sustrik <sustrik at 250bpm.com> wrote:

> Hi Ian,
>
> Yes. They are ACKs, same as those found in TCP, SCTP and whole bunch of
> other protocols.


I read them as slightly different - a TCP ack is basically 'don't resend
this segment', while in this case it was a 'don't send any more segments'. I
know that effectively TCP acks will do that, but only to within the window
size, so it's a kind of side effect. Either way, yeah :)

if there's some value in hit HWM generating an upstream pub
>> message on XPUB sockets. If you could signal the subscription set that
>> the connection used, it would be easy to build systems to automatically
>> spin up more downstream consumers
>>
>
> That doesn't make sense in case of PUB/SUB IMO. Messages are not
> load-balanced, rather passed to all peers, meaning that having more
> consumers doesn't help in handling the load in any way, it may even make it
> worse.
>

That's true throughout a tree, but almost always a PUB/SUB is going to
terminate in leafs of the tree that are actually *doing* something with the
messages - if those nodes can't do whatever it is fast enough then they're
going to consumer more slowly, which causes backpressure. At the moment this
is handled by having limited HWM and dropping extra messages, this was just
a general thought about having a smaller HWM, with the ability to make
improve the rate the queues drain at the leaves.


>  or throttle message production
>>
>
> The problem here is that this way a single slow/dead consumer can
> completely throttle the publisher and thus stop the flow of messages even to
> all the well behaved consumers.
>

True again, but I think there are a bunch of use cases where you could
throttle effectively without locking up the entire tree. For example, sensor
measurements for a variety of different sensors, you could sample less
frequently on specific ones if your processing network is unable to keep up.
The whole tree is only necessarily blocked if you have homogeneous data,
which is probably more common, but not necessarily true. I kind of think the
problem you mention in the SP post regarding buffer bloat is what I am
referring to - if you want mostly reliable delivery under normal conditions
but with variable load, you end up over allocating HWMs. If instead you
could signal rates through the system, you could tune publishing levels to
where you get better latency by sending somewhat less data.

Ian
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.zeromq.org/pipermail/zeromq-dev/attachments/20110928/e9506780/attachment.htm>