[zeromq-dev] Confirmation of ZMQ_PUB + ZMQ_SWAP intended behavior

[Martin Sustrik]

On 08/17/2011 11:50 PM, Pieter Hintjens wrote:
> On Wed, Aug 17, 2011 at 11:38 PM, Ian Barber<ian.barber at gmail.com>  wrote:
>
>> If there is an identity on ZMQ (<  4) SUB, then the publisher will buffer up
>> to the HWM. If the SWAP is set it will effectively extend that HWM up to
>> it's size. If there's no identity, the HWM wont be hit, and nothing will be
>> written to swap.
>
> You sure about this? I didn't realize the HWM was dependent on the SUB
> setting an identity...

There are multiple issues mixed here. Let's deconstruct the whole thing:

1. There's a buffer on SUB side as well as on the PUB side. The pushback 
works as follows:
    a. If sender application sends faster than the network is able to 
transfer, the buffer on the send side starts filling. When it reaches 
HWM it enters the exceptional state and starts dropping messages.
    b. If messages are arriving at receiving application faster than the 
application is reading them, the buffer on the receive side starts 
filling. When it reaches HWM it enters the exceptional state and stops 
reading from the underlying TCP connection.
    c. If b. happens, TCP buffers on both receive and send side start 
filling up and ultimately the sender application won't be able to send 
more data to the network (TCP pushback). At that point a. kicks in and 
ultimately results in dropping messages.

    You can thing of the algorithm as a series of tubes of various 
diameters. If there's too much liquid poured into the tubes to smoothly 
flow through a particular portion of the tube, the liquid starts to 
accumulate upstream from the critical portion of the tube and eventually 
all the upstream tubes become full of water and you won't be able to 
pour any additional liquid in.

2. SWAP semantics are opaque. The limit of the buffer is expanded by the 
size of the swap. In other words, backpressure kicks in only after HWM + 
SWAP is reached.

3. IDENTITY has no effect on the backpressure mechanism. The only thing 
it does is that the "connection" is not cancelled when the peer dies. 
I.e. at the moment the receive peer dies the messages start accumulating 
in the sending peer (ultimately reaching HWM+SWAP and applying 
backpressure) and start flowing only after the receiving peer is restarted.

As a side note, I would strongly suggest to set HWM to some sane finite 
value by default. Default infinite HWM is just confusing people. Compare 
TCP sockets where there's fixed default buffer size and nobody seems to 
have a problem with understanding the backpressure mechanism and 0mq 
with infinite default buffer size where the backpressure is constant 
source of confusion.

Martin