[zeromq-dev] PGM: multiple listener behavior? interface network syntax? was Re: ZMQ_MCAST_LOOP with PGM

[Stuart Levy]

On 8/2/12 6:20 AM, Pierre Ynard wrote:
>> The OS only forwards packets to the first open socket.
>> Check the RFC's on PGM and PGMCC for details of the protocols in
>> question,
>>
>> http://www.ietf.org/rfc/rfc3208.txt
>> http://www.ietf.org/proceedings/57/I-D/draft-ietf-rmt-bb-pgmcc-02.txt
> To be clear, if I understand correctly, OpenPGM binds a UDP socket
> to the port given with the multicast address, using SO_REUSEADDR.
> Then it uses this socket to both receive the multicast stream (hence
> SO_REUSEADDR) and the unicast NAKs. This is why if a sender and a
> receiver run on the same host, the socket of only one of them will
> receive unicast NAKs, breaking it for the other one. Right?
[...]

Hmm.  So can we tell what is the consequence of that, at ZMQ level, if 
NAKs get lost?
If a reliable-multicast receiver recognizes that a message was lost, 
sends a NAK,
and then *no* re-send follows (since the NAK was misdirected on 
receipt), then ...

    (a) does the ZMQ socket on the receiver struggle to keep messages 
in-order, so that it
        doesn't dare pass along later messages until missing ones have 
been successfully resent
        (which won't happen if NAKs are reliably lost)?  or ...
    (b) or, does the ZMQ-level receiver see a normal stream of messages, 
without much delay,
        but with a message missing?

For my purposes I'd much prefer (b), missing message with minimal delay, 
so that I can retry at application level.

I'm trying to build a distributed barrier -- N similar node processes, 
one per host,
which need to synchronize periodically.

To implement this I'm using PUB/SUB pairs, one pair per host,
with all N nodes publishing/subscribing to the same
epgm://interface;multicast-addr:port
address.

At the barrier, all nodes send a "node N is ready" message on PUB,
and all nodes listen on SUB to hear this msg from all their peers before 
they continue.
By using multicast I hope to avoid needing any central coordinator.

(ZMQ_MCAST_LOOP doesn't matter for this purpose, but multiple UDP 
sockets listening on the same unicast port does.)