[zeromq-dev] Multiplexing on top of TCP and subports

[Ian Barber]

On Thu, Oct 6, 2011 at 12:05 PM, Martin Sustrik <sustrik at 250bpm.com> wrote:
> On 10/06/2011 01:02 PM, Martin Sustrik wrote:
>
>> There are two subproblems to this AFAICS:
>>
>> 1. How do we communicate the "agreed upon message rate" throughout the
>> distribution tree?
>>
>> 2. How do we notify the applications that they are not adhering the
>> message rate contract?
>
> Ah. One more issue:
>
> 3. The network itself takes part in the contract, ie. it should be able to
> transfer the messages at the agreed upon rate. Who are we going to notify if
> it's not keeping up with the rate? Network admin presumably... But how
> exactly?
>
> Martin
>

For point 1, that depends on the question of where the message rate
comes from - do you have the rate controlled at the publisher (as with
PGM i guess), with the clients responsibility to keep up? That results
in something pretty similar to the current setup, but with the
addition of some sort of 'expected rate' message that could be
propagated to leaves of the distribution tree. Or do you have clients
advertise the rates at which they consume, and push that
responsibility back up? That was roughly where I was going with my
initial thought, but it is effectively just setting an window size
which would then need a flow of what are basically ACKs back up for
every message to add sending credit, which seems like a lot of
needless traffic even if you aggregate at each node.

For point 2, I think the  behavior of blocking on HWM would be better
in a 'rate exceeded' cases for publishers. For consumers, I'm not sure
how you could signal that they were consuming at too slow a rate,
unless you had explicit HWM hit messages - which itself gets back to
the problems that we discussed back at the unconf in brussels.

For point 3, the basic problem doesn't seem dissimilar to the TCP
congestion issue - you agree a maximum rate for flow control (window
size, which is really receiver controlled), but then actually have to
scale the window size up to that maximum during the communication in
order to test what the network can actually handle - if there's packet
loss, you scale back. That has the advantage that the network is
treated as a dumb resource - which would probably be the right general
behavior for PUB/SUB as well. This would need a method of adjusting
the rate throughout the distribution tree dynamically though!

I guess, if you had a separate nack channel (as in some of Pieter's
guide patterns), you could use somewhat similar logic:

Publishers have a publishing channel, a NACK channel, and a predefined
minimum rate, maximum rate, a rate increment, and clock interval
Publishers start publishing on the minimum rate
After each clock interval passes, publishers double their rate, until
the rate reaches maximum

Consumer read messages
New consumers read from the first message they receive - they do not
request old messages with the NACK channel
If a consumer does not get contiguous sequence numbers in message, it
sends a NACK to the publisher

When the publisher receives a NACK, it halves the message rate
After a clock interval with no NACK, having previously received one,
it increases the rate by one rate increment per clock interval

NACKs could be sent up the chain to allow centrally adjusting rate.

This could also be built easily on top of existing zmq pubsub.

Ian