[zeromq-dev] First draft for PPPP rfc (paranoid pirate publish protocol)

Goswin von Brederlow goswin-v-b at web.de
Thu Aug 28 17:22:26 CEST 2014

On Thu, Aug 28, 2014 at 06:40:10AM -0400, André Caron wrote:
> Interesting spec :-)
> The one thing I might be concerned about is the high volume of bandwidth
> required by ACK messages.  The're a section in the guide that does a quick
> computation of the bandwith requirements for heartbeating and the
> conclusion is to treat any message from a peer as a heartbeat because it's
> as effective and isn't as prohibitive (indeed, that's what your spec
> recommends for heartbeats).  It might be worth coming up with a quick
> bandwith estimate using the same type of formula for ACK messages.
> If I may, I'd recommend storing the last ACK-ed sequence number in all
> messages sent to a peer.  This assumes your application cannot process
> messages out of order, but it completely eliminates the need for ACK
> messages: you will automatically encode the ACK into the next
> message/reply/error reply you send or in the next hearbeat.
> Any thoughts on this?
> Cheers,
> André

That is what I'm doing. Look at "Message Specifications". All messages
contain an ACK sequence number and optionally an array of NACK
sequence numbers. So as long as there is a steady send and recv of
messages the ACK will go with that. If no message is send then the
next heartbeat will carry the ACK.

The overhead for ACKs is 8 byte sequence number + 1 byte length for
the zmq on-wire encoding. Another 9 bytes for the message sequence
number, 2 bytes for the flags, 1-4 + 8 * <num NACKs> bytes if there
are NACKs.

So every message (or heartbeat) is at least 20 byte larger. Some more
with NACKs but that should be rare.


With tcp NACKs can only occur when a message is lost during transport.
I think that is limited to a connection disconnect/reconnect while the
message is in the air. In that case everything in the kernels socket
buffers is lost.

In the specs I wrote:
   The ACK sequence number is the highest integer so that all message
   sequence number less or equal where received without gap (excluding
   gaps listed in NACK, see below).

I changed my mind there while implementing. Say I do have NACKs and I
also have 10 messages to send. Do I then send one message with NACKs
and a high ACK and the next 9 without NACK and a lower ACK? Sending
the NACKs 10 times seems wastefull. But I don't want the ACK counter
to ever fall.

It seems better for the ACK to be the highest number up until the first
gap. The optional NACKs are then all larger than the ACK. The peer can
then decide wether to send NACKs or not on a message per message basis
and the ACK sequence number is strictly monotonic.

The drawback is that the sending peer can not destroy messages
received out-of-order by the peer. If there is a gap in the stream
then all messages after the gap must stay cached till the gap can be
closed. Maybe everything below the highest NACK that isn't itself an
NACK could be destroyed but the complexity probably outweighs the gain.


More information about the zeromq-dev mailing list