[zeromq-dev] Any way to get info on subscribers?

Brian Granger ellisonbg at gmail.com
Sat Jul 24 18:43:48 CEST 2010


On Sat, Jul 24, 2010 at 9:01 AM, Martin Sustrik <sustrik at 250bpm.com> wrote:

> Brian,
>
>
>  I really like the idea of sticking to the connectionless model of 0MQ.
>>  It takes a while to get used to, but we are finding that when we are
>> forced to design without connection information, we write better apps.
>>
>
> Yes. That's the idea in the background: Enforcing scalable messaging
> patterns.
>
> Adding functionality like "get number of subscribers" breaks the
> scalability. You can use it as far as you need no devices in the middle.
> Once your business grows and you add devices to scale your application up
> the code using this feature would silently break.
>
> The real problem here is that people have their applications written in
> non-scalable way and at the same time want to use 0MQ to replace their
> existing messaging infrastructure. The options are:
>
> 1. Forget about 0MQ, use BSD sockets (or something else) instead.
> 2. Refactor your app in a scalable way.
> 3. Add non-scalable features to 0MQ.
>
> As 1 or 2 is often not an option, the idea proposed by Pieter (splitting
> 0MQ into two pieces -- lower layer with no guaranteed scalability, but some
> useful features such as transport abstraction etc. and upper layer allowing
> only for scalable messaging patterns) makes sense.


I think there is a 4th option that I tend to prefer:

4.  Implement the needed features (such as connection focused API and
presence features) on top of the scalable 0MQ core.  Here are some ideas...

We are finding that one of the keys to using 0MQ effectively is to stop
trying to replace each single TCP connection with a single 0MQ socket.
 Instead, the following mapping is much more useful:

1 connection oriented TPC socket ==> multiple 0MQ sockets bundled together

Here is a diagram of a system that we are using this approach for:

http://github.com/ipython/ipython/commit/e21b32e89a634cb1393fd54c1a5657f63f40b1ff

We used to be using a single TCP socket for this.



>
>   But, we do end up creating applications that use other 0MQ sockets to
>> communicate presence information.  To handle the case described here
>> (knowing what SUBs are connected to a PUB), we would probably do:
>>
>> * Have the publisher also have an XREQ socket that the SUBS and use an
>> XREP to notify the publisher of its presence.
>> * For high availability/fault tolerance, give each subscriber a
>> heartbeat and have the publisher monitor those to detect subscriber
>> failures.
>>
>> While it take a bit to write this type of code, we find it works
>> extremely well and honestly easier than handling connection logic in
>> traditional TCP sockets.
>>
>
> I think the main use case here is that people often want two distinct
> functionalities:
>
> 1. "production subsystem" that handles the real work
> 2. "administrative subsystem" that keeps track of different components
>
> (see attached diagram)
>

One thing that we did recently was to implement a 3 socket device that is
basically a queue device with an additional monitoring socket that allows an
administrator to monitor the device.

Cheers,

Brian



> These two, IMO, should be clearly separated. However, with most legacy apps
> this is not the case.
>
> Martin
>
>


-- 
Brian E. Granger, Ph.D.
Assistant Professor of Physics
Cal Poly State University, San Luis Obispo
bgranger at calpoly.edu
ellisonbg at gmail.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.zeromq.org/pipermail/zeromq-dev/attachments/20100724/90107b45/attachment.htm>


More information about the zeromq-dev mailing list