[zeromq-dev] Slow joiner syndrome solution for XSUB/XPUB broker based system

[Gyorgy Szekely]

Hi Tomer
As far as I know the message from the publisher will reach the broker.
According to the docs, the PUB socket drops messages in mute-state (HWM
reached), and it's not the case here. The message will be sent as soon as
the connection is established, and the socket termination blocks until the
send is complete. Unless you set linger to zero.

The slow joiner problem means that subscriptions may not be active by the
time the publisher send the message. Either because the subscriber is not
yet running, or because the subscribe calls themselves are asynchronous (by
the time setsockopt(SUNSCRIBE) returns the broker is not yet aware of
this). The zmq guide shows mitigations for this problem in the Advanced
Publish Subscribe chapter.

Regards,
  Gyorgy

On Mon, May 28, 2018 at 11:06 AM, Tomer Eliyahu <tomereliyahu1 at gmail.com>
wrote:

> Hi,
>
>
>
> I know this topic was probably discussed before, I couldn't find a proper
> solution, so I implemented something a bit different. I'm not sure if this
> solves all pitfalls, i'll be greatfull for comments.
>
>
>
> We have a system with a XPUB-XSUB broker running as a separate process in
> the system (binds frontend to ipc:///tmp/publishers  and backend to
> ipc:///tmp/subscribers).
>
>
>
> Clients of the broker have both SUB socket for receiving messages, and a
> PUB socket for sending messages. When a client boots, it connects both its
> PUB and SUB sockets to the broker's endpoints, and subscribes to the topic
> of interest.
>
>
> For the sake of simplicity, lets assume there we have only the broker, a
> publisher and a subscriber processes in the system:
>
> We make sure that the broker process starts first, then a subscriber which
> connects and subscribes to the topic, and only then start the publisher.
> The publisher then sends a single message and terminates.
>
> Obviously, the message is lost due to the slow joiner syndrome - I assume
> the reason for that is because the publisher process zmq_connect() call is
> asynchronous, therefore the connect is not actually complete by the time we
> send the message.
>
>
>
> I thought of a possible solution for this - basically we want to
> synchronize the connect operation done by the publisher. Having both PUB
> and SUB socket, we can simply send a dummy message from PUB to SUB on the
> same publisher process until the first message is receieved, and then it is
> guarantied that the connect is done and consecutive messages (now to "real"
> topics with actual subscribers) will not be lost.
>
>
>
> The only part i'm not sure about is the subscriber side - assuming the
> subscriber boots, connects and subscribes _before_ we start the publisher -
> is it guarantied that no message will be lost (assuming ofcourse the
> subscriber doesn't crash / unsubscribe / etc.) ?
>
>
>
> Thanks,
>
> Tomer
>
>
> _______________________________________________
> zeromq-dev mailing list
> zeromq-dev at lists.zeromq.org
> https://lists.zeromq.org/mailman/listinfo/zeromq-dev
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.zeromq.org/pipermail/zeromq-dev/attachments/20180528/f92db7b1/attachment.htm>