[zeromq-dev] Router Example & questions

Mon Mar 10 04:02:49 CET 2014

Pieter,

Thanks for the conversation, it's getting me thinking.  I am reading through
Chapter 8 again to be sure that I'm not missing anything, and this paragraph
stood out:

> In a world of trillions of nodes, the ones you talk to most are the
> ones closest to you. This is how it works in the real world and it's
> the sanest way of scaling large-scale architectures. Groups of nodes,
> logically or physically close, connected by bridges to other groups
> of nodes. A local group will be anything from half-a-dozen nodes to a
> few thousand nodes.

Now, I can't discount or refute this.  I can only tell you that I'm trying
to solve a different problem.  I want to communicate with nodes far and
wide.  At the lowest level of the overlay network, I want nodes establishing
peering with other nodes from as far away as possible, which will actually
make the overlay network smaller, reducing the total number of hops for a
message to reach from any point to any other point.

With regards to discovery, your beacon is awesome for networks confined to a
single broadcast domain.  Indeed, Cisco does this with CDP, a UDP beacon
that's also used by other platforms such as Mikrotik and Ubiquiti.
Unfortunately, the beacon is probably unsuitable for nodes discovery across
the vast reaches of space and time.  I'm thinking a 3 stage bootstrapping
process is possibly the best approach.  Starting with the use of DNS seeds,
querying those seed nodes, and finally advertising on the overlay network
for new peers (general or for a specific topic).

To facilitate this, nodes need to track address information for their peers,
and while ZMQ_SRCFD helps facilitate this at a higher level, my line of
thought is that it could be done easier and cheaper if it was done at the
lower level and made available without an API call.  If you could tell 0MQ
to use the peer's IP information in the message envelope, problem solved
with just a few more bytes of data.

With all that said, I'm still not sure that 0MQ in general, or the ROUTER
specifically, is the right tool for the job, but between its ease of use and
your prose, I really hope so!  0MQ is by far, the best experience I've ever
had working with sockets.

At any rate, I'll continue plugging away with what I have and see where I
end up.

Thanks again!

-Troy

-----Original Message-----
From: zeromq-dev-bounces at lists.zeromq.org
[mailto:zeromq-dev-bounces at lists.zeromq.org] On Behalf Of Pieter Hintjens
Sent: Sunday, March 9, 2014 5:14 PM
To: ZeroMQ development list
Subject: Re: [zeromq-dev] Router Example & questions

The design from Chapter 8 is the simplest and most robust one I've found.

Identifying a peer by IP address isn't great. What if you want to run
100 peers on the same IP address? You should use a identifier that's
independent of physical address. On some networks (WiFi), a peer can be
disconnected and then reconnect with a different IP address, but should
continue to run.

-Pieter

On Sun, Mar 9, 2014 at 9:40 PM, Troy Settle <troy.settle at gmail.com> wrote:
> Pieter,
>
> Thanks for the reply.  The problem I have with the dealer, is the 
> round-robin style of sending messages.  The fix given in Chapter 8 to 
> use one dealer per peer is workable, but would seem to complicate 
> things too much.
>
> I'm not sure I see the fragility introduced by the ROUTER-ROUTER pair, 
> but this is something that would be handled by the application.  If N1 
> sends a request to N2 and does not receive a reply, that's ok... we 
> want to be as tolerant of lost peers as possible.
>
> The pattern I envision, is that every node is running identical code 
> with identical capabilities.  To keep the underlying protocol handling 
> as simple as possible, the sockets should be identical on both sides 
> of every connection.
>
> Any thoughts on the other part of my message regarding the peer identity?
> Is an part doable without incurring additional expense in the library?
>
> Thanks!
>
> -----Original Message-----
> From: zeromq-dev-bounces at lists.zeromq.org
> [mailto:zeromq-dev-bounces at lists.zeromq.org] On Behalf Of Pieter 
> Hintjens
> Sent: Sunday, March 9, 2014 4:12 PM
> To: ZeroMQ development list
> Subject: Re: [zeromq-dev] Router Example & questions
>
> Hi Troy,
>
> There's a pattern for fully decentralized clusters, in Chapter 8. To 
> ensure we don't lose messages randomly, we use dealer-router in a
symmetric way.
> Pure router-router is fragile in several ways, and I'd not recommend it.
>
> -Pieter
>
> On Sun, Mar 9, 2014 at 8:20 PM, Troy Settle <troy.settle at gmail.com> wrote:
>> I've been digging around 0MQ for a short time now, and one of the 
>> things that is not discussed heavily in the guide, is the ROUTER to 
>> ROUTER pair, which would be highly useful in a completely 
>> decentralized P2P network.  So, after working through some of the 
>> examples in the Guide, I came up with this code.  It's in PHP, and if 
>> nobody takes my fun away, I'll write it also in C and/or C++ 
>> (assuming I can get back up to speed in them).  Please feel free to 
>> use this in the
> 0MQ Guide (with or without cleanup).
>>
>> http://pastebin.com/zW548wpN
>>
>> I called it node.php.  There's probably a better name as a piece of 
>> example code.
>>
>> $ php node.php <port> <peers>
>>
>> It will fork out as many peers as you want (at least 2), each binding 
>> to <port>+n and connecting to each previous node that's already bound.
>> What you end up with, is any number of nodes, all connected (in some
>> way) to all the other nodes.  If you launch a few nodes, all messages 
>> will
> be received.
>> If you launch many nodes, messages will dropped, as the oldest and 
>> youngest nodes don't overlap quite enough.  You can play with the 
>> sleep on line 60 to have some fun.  Comment out the printf() and echo 
>> statements in the loop if you want to reduce the noise.  The 
>> zhelpers.php include isn't necessary, it was left over from a 
>> previous
> experiment.
>>
>> My test machine is a single CPU VM running FreeBSD 10 and PHP 5.5.9.
>> The largest test I ran was 100 nodes which was much fun to watch.
>>
>> ---
>>
>> One thing that I find seriously lacking, is the ability to identify a 
>> peer by IP address.  I've seen the recent patch for ZMQ_SRCFD, which 
>> will be handy in C and presumably in other languages as patches are 
>> made to the bindings.  I'm wondering, though, if the ROUTER and 
>> STREAM sockets couldn't be configured (perhaps by default) to 
>> providing some useful information in the identity string such as the 
>> descriptor, address family, address, port, and perhaps topic.
>>
>> One other thing that I'm wondering, if there couldn't be a low-level 
>> broadcast/flood mechanism built into the ROUTER/STREAM socket.  Sure, 
>> it's easy enough to iterate through all peers, but if I want to send 
>> the same message to several hundred peers, doing so at the lowest 
>> level in the 0MQ API should make the operation much less expensive.
>> Of course, not all peers will want to receive floods, so we would 
>> probably
> need a flag for that.
>> Perhaps the first byte of the identity string could be used for flagging.
>>
>> 0x00 - Random 32bit ID  (current)
>> 0x01 - Peer generated ID (minor change)
>> 0x02 - Socket Info ID (added functionality)
>> 0x04 - Can receive floods (capability flag) ...
>> 0xFF - Broadcast a flood (sending only)
>>
>> All in all, I'm not even close to being up to speed in C/C++ to do 
>> this myself, mostly I'm just thinking about ways to extend 
>> functionality without incurring too much cost and hoping to catch 
>> someone's attention.  We would probably need much discussion to flesh 
>> these ideas out if it is to go anywhere at all.
>>
>> -Troy
>>
>>
>> _______________________________________________
>> zeromq-dev mailing list
>> zeromq-dev at lists.zeromq.org
>> http://lists.zeromq.org/mailman/listinfo/zeromq-dev
> _______________________________________________
> zeromq-dev mailing list
> zeromq-dev at lists.zeromq.org
> http://lists.zeromq.org/mailman/listinfo/zeromq-dev
>
> _______________________________________________
> zeromq-dev mailing list
> zeromq-dev at lists.zeromq.org
> http://lists.zeromq.org/mailman/listinfo/zeromq-dev
_______________________________________________
zeromq-dev mailing list
zeromq-dev at lists.zeromq.org
http://lists.zeromq.org/mailman/listinfo/zeromq-dev