[zeromq-dev] QoS support

[Martin Sustrik]

Hi Steven,

> There are many types of Quality of Service:
> 
> 1,  Packet level tagging with the ToS bits.  Only used for determining 
> traffic precedence at gateways, such as VoIP in preference to HTTP in 
> order to maintain low latency.

Right. I believe it is useful to have this level of control on the level 
of individual 0mq message streams. If all the traffic was passed through 
a single connection, it wouldn't be possible.

Actually, the only sane scenario for merging all the feeds into a single 
connection is the case where there's only one port open in the corporate 
firewall. And even in that case it wouldn't be that difficult to add a 
feature to 0mq that would use single dest port and distinguish between 
individual message streams using src port.

> Many OS stacks, NICs, switches, etc, completely ignore ToS as it is 
> slower to process the bits than perform best effort for every packet. 
>  There's been a long conversation on LKML about this.
> 
> Adding a host firewall to tag and interpret ToS tags is going to 
> introduce additional latencies for every packet.

True.

> There may be a fundamental problem with retransmission handling at the 
> transport layer depending on which protocol is being used.  Similarly 
> with an ordered transport what happens if you send a high latency 
> message followed by a low latency message.

That's exactly the reason why I wanted QoS moved as far down the stack 
as possible.

When large low priority message is followed by small high priority 
message, single TCP connection would mean that the latter will be 
blocked till the former is transferred ("head of line blocking").

Using two TCP connections, one for each priority, minimises the blocking 
to the level where at most MTU (1500) bytes of the low priority message 
are sent before high priority message gets through.

> 2,  Interpretation of quality to mean the age of the data, i.e. 
>  realtime, delayed, or conflated.
> 
> 3,  Traffic priority at the application level, individual messages in a 
> stream are tagged with different priorities and each receiving 
> application needs to filter the incoming stream to the different 
> priority queues.
> 
> 4,  Traffic priority at the receiving application, the receiver tags 
> certain topics with different priorities based on it's interest and 
> filters the incoming stream appropriately.

Pts 3 and 4 are the case with most traditional messaging servers - use 
single connection, sort the messages based on priorities on the 
endpoints. Obviously, head-of-line blocking results. Actually this was 
one of the problems that made me start thinking of an alternative 
architecture and ultimately developing 0mq.

> 5,  Priority streams, if you have 3 priority levels create 3 transports 
> using 3 known ports and have routers prioritise at the port level.

> So the question of choice is dependent on whether every application 
> using the internet routes is under your control to set the ToS bits and 
> how do you expect it to perform better than simply prioritizing separate 
> transports.

Right.

Martin