[zeromq-dev] Can a pub-sub endpoint have multiple publishers?

[Pieter Hintjens]

Looking at the ZMQ code will not help you do what you need, though it
may be useful for other reasons.

First off, the rules are different if you're using a TCP based pub-sub
model, vs. a multicast model like PGM (or NORM, I think, but am not
familiar with that yet).

Second, if you use e.g. PGM then yes, you can clearly do N-to-N
publish-subscribe using a single endpoint. Your switch is acting as
the equivalent of a broker.

If you do not want to use a broker and you are using TCP then you can
interconnect subscribers and publishers. That's doable and fair and
fast enough.

As a general rule when asking for help here, tell us specifics about
your case, numbers and volumes and sizes. Then work through the Guide,
don't just read it, so we are speaking the same language. Then, make
minimal proofs of your core flow, and come back to us with issues you
find on those.

"I'm new to ZeroMQ so help me with my problem and if not, I'll fix the
code myself" is not productive, though I personally find that quite
amusing.

-Pieter

On Mon, Apr 14, 2014 at 10:48 PM, James Renkel <james at mfluent.com> wrote:
> Randall,
>
> Our application does not require 100% reliability (That, of course, is
> theoretically impossible!).
>
> The reliability level that PGM, EPGM, NORM provide is all we're looking
> for, along with as simple an implementation as possible.
>
> So I guess we're going to have to do something to get it.
>
> Maybe I'll look at the ZMQ code and see if that can be added without
> requiring the equivalent of message brokers. Since you needed it and we
> needed it, maybe others do, too.
>
> Thanks for your reply.
>
> Jim Renkel
> Senior Technical Staff
> mFluent, Inc. LLC
> mailto:james at mfluent.com
>
>
> On 14/4/14, 15:38 , Randall Nortman wrote:
>> You're correct, my implementation has a single point of failure, which
>> is fine in my application.  You could improve upon this by running
>> multiple forwarding daemons, but all clients will need to "know about"
>> all the daemons so they know where to connect to, where to fail over
>> to, how to detect failures, etc.  There are multiple options for a
>> setup like this, but nothing that works out of the box as far as I
>> know.
>>
>> Note also that if you need 100% reliability (that is, you need to know
>> that when you send a message it absolutely, definitely reaches all
>> listeners), then you are also venturing beyond what ZMQ offers out of
>> the box (for good reasons).  Reliable broadcast/fan-out is a difficult
>> problem and there's no one-size-fits-all solution.  There are some
>> reliable pub-sub patterns in the Guide.  All involve more complexity
>> than just PUB and SUB sockets.
>>
>> On Mon, Apr 14, 2014 at 03:29:19PM -0500, James Renkel wrote:
>>> Randall,
>>>
>>> Thank you for your reply.
>>>
>>> As I understand your answer, we would have to provide out own daemon.
>>>
>>> But that becomes a potential single point of failure, so it would need
>>> to be replicated.
>>>
>>> So, don't we then wind up back at the same place and still have the
>>> problem that both multiple publishers and multiple subscribers must each
>>> connect to multiple daemons?
>>>
>>> Or am I missing something here?
>>>
>>> Jim Renkel
>>> Senior Technical Staff
>>> mFluent, Inc. LLC
>>> mailto:james at mfluent.com
>>>
>>> On 14/4/14, 15:05 , Randall Nortman wrote:
>>>> I had the same requirement in my application.  You need a forwarding
>>>> device (either using the official ZMQ device or your own tiny bit of
>>>> code) to make this work.  Yes, you can have multiple PUBs connected to
>>>> a single SUB, but there is no way out of the box to connect multiple
>>>> PUBs to multiple SUBs.  So you have a daemon that binds both a PUB and
>>>> a SUB.  Senders connect a PUB to the daemon's SUB socket, and
>>>> receivers connect a SUB to the daemon's PUB, and the daemon just
>>>> forwards everything from its SUB to its PUB.
>>>>
>>>> Instead of a SUB on the daemon, you can just as easily have a PULL
>>>> socket, and then the senders have a PUSH.  It's really pretty much the
>>>> same thing.  You can even do this with ROUTER/DEALER, but that offers
>>>> no advantages.  You must use a PUB socket on the daemon's send side,
>>>> because there is no other socket type that does fan-out (deliver one
>>>> message to multiple recipients).
>>>>
>>> _______________________________________________
>>> zeromq-dev mailing list
>>> zeromq-dev at lists.zeromq.org
>>> http://lists.zeromq.org/mailman/listinfo/zeromq-dev
>>>
>> _______________________________________________
>> zeromq-dev mailing list
>> zeromq-dev at lists.zeromq.org
>> http://lists.zeromq.org/mailman/listinfo/zeromq-dev
>
> _______________________________________________
> zeromq-dev mailing list
> zeromq-dev at lists.zeromq.org
> http://lists.zeromq.org/mailman/listinfo/zeromq-dev