[zeromq-dev] Any way to get info on subscribers?

[Brian Granger]

On Sun, Jul 25, 2010 at 7:36 AM, Steven Clark <steven.p.clark at gmail.com>wrote:

>
>> > I think there is a 4th option that I tend to prefer:
>> >
>> > 4.  Implement the needed features (such as connection focused API and
>> > presence features) on top of the scalable 0MQ core.  Here are some
>> ideas...
>> >
>> > We are finding that one of the keys to using 0MQ effectively is to stop
>> > trying to replace each single TCP connection with a single 0MQ socket.
>> >  Instead, the following mapping is much more useful:
>> >
>> > 1 connection oriented TPC socket ==> multiple 0MQ sockets bundled
>> together
>>
>> WOW!!! You've put down the basic principle that nobody explicitly
>> expressed so far! It should be put at the beginning of any 0MQ
>> documentation in BIG RED LETTERS.
>>
>> Let's think about how to convey the idea to people struggling to port
>> their existing architectures to 0MQ...
>>
>> >     I think the main use case here is that people often want two
>> >     distinct functionalities:
>> >
>> >     1. "production subsystem" that handles the real work
>> >     2. "administrative subsystem" that keeps track of different
>> components
>>
>
> Original poster here. Thanks for all the replies. I understand the concept
> that 0MQ supports multiple transports, and that a notion of "connectedness"
> only makes sense for some of them. However, I "know" that I am using TCP
> (since I'm the one that created the bind string, of course). It seems
> unfortunate that I am limited to functionality in the lowest common
> denominator of all the transport types.
>
> My use case is as follows: pretend there are two PUB sockets. One
> broadcasts telemetry from NASA Rover A, one broadcasts telemetry from Rover
> B. Each Subscriber is a technician that is observing a telemetry stream. I
> want to be able to provide the technicians with a list of who is watching
> each of the Rovers, such that the technicians could (if they choose) spread
> themselves more evenly among the two Rovers, or perhaps call more people in
> to work if there aren't enough eyes on a given Rover.
>
>
You are probably going to need 1 or more central processes (let's call it a
broker for now) that manages the system for you.

First, the PUB/SUB side of the picture:

The BROKER will have 1 SUB socket that the ROVERs' PUB sockets connect to.
 ROVER's will publish to their PUB socket using topics that include an
unique ROVER identity.

The BROKER will have 1 PUB socket that republishes the ROVERs' published
messages.

Before forwarding the ROVERs' messages, the BROKER will introspect the
message to see what ROVERs are publishing messages (they will just look at
the topics that include that).  This will allow the BROKER to track which
ROVERs are currently publishing messages.

The BROKER will have an XREP socket that TECHNICIANs will use to
register/unregister that they are watching a topic (ROVER).

TECHNICIAN can also use the XREP socket to get a list of currently published
topics and which TECHNICIANs are subscribed to which topics.

The TECHNICIANs will have a SUB socket that will subscribe to the topics of
the BROKER.  When a TECHNICIAN subscribes to a topic, it uses a XREQ socket
to tell the BROKER that it is watching that topic (remember TOPICS represent
a ROVER).

I was hoping to accomplish this with 2 PUB sockets, and N SUB sockets, but I
> guess it jus' ain't that easy? What pattern of 0MQ sockets makes sense here?
> (A further wrinkle: I want at most one of the N Subscribers to have
> "command" capability to a given Rover. I was planning on making this take
> place on a separate PAIR channel -- is this a good way?)
>
>
I would have the control channel be a separate CONTROL BROKER process that
both the ROVERs and TECHNICIANs connect to.  We have found that it is
important in 0MQ to cleanly separate control (request/reply) from
observation (PUB/SUB).  When you try to mix the two together, you get a
mess.  But, the CONTROL BROKER can use its down set of sockets and the
TECHNICIANs and ROVERs will use appropriate matching sockets.  PAIR could be
a good option, or possibly look at using inverted XREP/XREQ between the
ROVER and CONTROL BROKER.  IT is not quite clear to me which is the best
approach.

In my experience, it really helps to start prototyping things using a high
level language such as Python (I am biased towards Python, but any high
level langauge will do).  That will enable you to quickly try out different
socket combinations.

I agree that I need to have a mental-paradigm-shift to make better use of
> 0MQ. Brian's comment that "1 connection oriented TPC socket ==> multiple
> 0MQ sockets bundled together" seems strange...isn't the role of
> middle-ware usually to reduce complexity/reduce the number of things I need
> to create to get the job done?
>

That is the big surprise.  It *will* reduce complexity, even though it won't
seem like ti at first.  Try to go implement the architecture you are
describing using TCP sockets (OK, maybe you have).  You will find that to
support all the features you have described, you will write more code than
you do using 0MQ, it will have worse performance and be harder to maintain.
 You will basically end up rewriting parts of 0MQ yourself (you will need
queues to make things properly asynchronous).  In my experience, once you
find the right 0MQ socket combination, the amount of code you will need is
very minimal.

Sounds like a fun project though.

Cheers,

Brian


> Thanks again for the discussion.
> -Steven
>
> _______________________________________________
> zeromq-dev mailing list
> zeromq-dev at lists.zeromq.org
> http://lists.zeromq.org/mailman/listinfo/zeromq-dev
>
>


-- 
Brian E. Granger, Ph.D.
Assistant Professor of Physics
Cal Poly State University, San Luis Obispo
bgranger at calpoly.edu
ellisonbg at gmail.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.zeromq.org/pipermail/zeromq-dev/attachments/20100725/b8511ef1/attachment.htm>