[zeromq-dev] Dafka towers

[Bill Torpey]

Hi All:

On a related note, we use a similar approach to discovery in OZ (https://github.com/nyfix/OZ/blob/master/doc/Naming-Service.md <https://github.com/nyfix/OZ/blob/master/doc/Naming-Service.md>), with some differences:

+ OZ also uses UUIDs to identify nodes
~ In OZ, the “towers” are simply 0mq proxies (sometimes referred to as “nsd” for historical reasons).  They store no state and have little code (other than a call to zmq_proxy) but simply pass messages along: https://github.com/nyfix/OZ/blob/master/src/nsd.c <https://github.com/nyfix/OZ/blob/master/src/nsd.c>
- OZ doesn’t have the concept of “stores”  — nodes are just nodes.
- An OZ node uses the proxy’s “welcome message” (https://somdoron.com/2015/09/reliable-pubsub/ <https://somdoron.com/2015/09/reliable-pubsub/>) to trigger publishing its discovery information (https://github.com/nyfix/OZ/blob/master/doc/Wire-Formats.md#naming-messages <https://github.com/nyfix/OZ/blob/master/doc/Wire-Formats.md#naming-messages>)
+ OZ also uses beaconing in a “belt-and-suspenders” approach to ensuring that there are no “orphan” nodes.  (In other words, in case the startup protocol has a race condition).

We’re looking to see if it’s possible to eliminate the beacon messages — perhaps using a tool like TLA+ to prove that the startup protocol is sufficient.  

We’d be very interested in any comments from the community on the discovery protocol specifically, or on OZ in general.  You can find out more here: https://github.com/nyfix/OZ <https://github.com/nyfix/OZ>

FWIW, OZ has been live since March 2020, processing ~ 50MM messages/day in the NYFIX Marketplace — the worlds’ largest FIX order-routing network.

Best Regards,

Bill 


> On Jan 21, 2021, at 10:00 AM, Adriano Ribeiro <adribeiro at gmail.com> wrote:
> 
> Hi Kevin,
> 
> Is this new? I completely missed that the rfc page is open for quiet time, so i did not see your update,I will look at and back to you with more questions. :) 
> 
> Att,
> 
> Adriano Ribeiro
> 
> 
> On Thu, Jan 21, 2021 at 9:52 AM Kevin Sapper <kevinsapper88 at gmail.com <mailto:kevinsapper88 at gmail.com>> wrote:
> Hi Adriano,
> 
> Nice that you're trying to implement a netmq version :)
> 
> Have a look at the rfc section for node discovery https://rfc.zeromq.org/spec/46/#node-discovery-and-presence <https://rfc.zeromq.org/spec/46/#node-discovery-and-presence>.
> 
> Does that help?
> 
> //Kevin
> 
> Am Do., 21. Jan. 2021 um 03:11 Uhr schrieb Adriano Ribeiro <adribeiro at gmail.com <mailto:adribeiro at gmail.com>>:
> Hi guys,
> 
> I'm trying to implement a netmq-based version of dafka, but I have some doubts about network bootstrap via beacons and towers.
> 
> I do not understand how to connect producers and subscribers to towers.
> 
> Any help would be appreciated :)
>  
> Att,
> 
> Adriano Ribeiro
> _______________________________________________
> zeromq-dev mailing list
> zeromq-dev at lists.zeromq.org <mailto:zeromq-dev at lists.zeromq.org>
> https://lists.zeromq.org/mailman/listinfo/zeromq-dev <https://lists.zeromq.org/mailman/listinfo/zeromq-dev>
> _______________________________________________
> zeromq-dev mailing list
> zeromq-dev at lists.zeromq.org <mailto:zeromq-dev at lists.zeromq.org>
> https://lists.zeromq.org/mailman/listinfo/zeromq-dev <https://lists.zeromq.org/mailman/listinfo/zeromq-dev>
> _______________________________________________
> zeromq-dev mailing list
> zeromq-dev at lists.zeromq.org
> https://lists.zeromq.org/mailman/listinfo/zeromq-dev

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.zeromq.org/pipermail/zeromq-dev/attachments/20210121/7c2df637/attachment.htm>