[zeromq-dev] ZRE/Zbeacon improvement proposal

[Pieter Hintjens]

On Sat, Oct 26, 2013 at 10:53 AM, Arnaud Loonstra <arnaud at sphaero.org> wrote:

> 1 - Only one node can run on a host (or one per interface)
> 1 - The issue is caused by using a broadcast socket. It's possible to
> bind multiple programs to the same address and port but packets are then
> load balanced between the sockets.

Is this right? My understanding (and test results) were that when
multiple listeners bound to the same socket, they all received the
incoming messages. The Zyre load tester uses dozens of nodes on the
same host:port. They're definitely all getting each others' packets.

> 2 - Network segments can't be covered

This is indeed a major lack.

> 3 - Nodes are trying to connect to dead peers (peers that are gone)

This is normal in some senses. ZRE was designed for WiFi and under
load, clients can come and go randomly. To get a resilient network you
need to be optimistic about connecting, and slower to discard peers as
"dead".

Multicast would be fine; the Android issue is afaik already solved in
Zyre through the tactic of double handshaking on the TCP connection.
I.e. if A can see B, and B cannot see A, then A will connect over TCP
to B and then B will reconnect back to A over TCP (not having seen it
before).

We can make multicast a configurable option on beacons. I'd not change
the existing default since it'll break applications.

I'm not so sure about using multicast to cross segments; it might be
better to do this explicitly with forwarding, i.e. when a node has two
network interfaces, it bridges traffic between them.

> 3 - What I observed was nodes not knowing when a peer node was gone
> trying to connect to the peer node. This is obviously to no success. It
> also results in unnecessary traffic and handling. I would propose to
> embody a node state identifier inside the beacon. This way when a node
> exits it can send its exit state before terminating and other nodes have
> the possibility to pick this up. This would also be a welcome feature to
> inform nodes of each other state. For example when a node is overloaded
> it could inform about this using its state. A broker can leave him alone
> for a while. I think this could be useful in general.

UDP broadcasts / multicasts are not reliable and are the first things
to be lost when the network is stressed (which is when you get client
disconnections).

It would not be wise to try to use these for state propagation.

Please consider zbeacon within the context of a full protocol such as
ZRE, which builds a TCP cluster on top of the zbeacon discovery. It
would be a shame to start mixing abstractions.

> - For passing multiple network segments multicast is often not possible.
> Crossing internet using multicast is nowhere available, to my knowledge.

Indeed. I do not like using multicast for its ability to leak across
networks; that's so hard to get right and can lead to such a mess.
Much better IMO to see 1-segment UDP as one mechanism for discovery,
tied together with internetwork discovery over TCP (which would work
across the Internet).

One step at a time. Could you think about not adding state to beacons
and instead looking at how ZRE does its interconnect. This will also
help you understand how to do other forms of discovery (e.g. an
application could simply tell a node, via the API, "here is a node at
hostname:port".

-Pieter