[zeromq-dev] Proxy with inproc frontend

Cosmo Harrigan cosmo.harrigan at singularityu.org
Mon Feb 3 15:57:11 CET 2014


Hi,

I have a scenario where I want multiple simultaneous event handlers to
publish ZMQ messages on one PUB socket. To do so, I used the following
design:

Create two sockets, call them sockA (XSUB) and sockB (XPUB)
Bind sockA to the inproc endpoint
Bind sockB to the tcp endpoint
Start a proxy forwarder device with sockA as the frontend, and sockB as the
backend

Whenever an event occurs, an event handler thread is created, in which a
socket is created; call it sockWn
Connect sockWn to the inproc endpoint
(***)
Send one message on sockWn
Close sockWn

The problem is, the proxy device loses almost all of the messages, unless I
insert a sleep statement where I marked (***).

Reviewing the guide and past forum posts, I think this is due to the slow
joiner syndrome, where the XSUB sockA doesn't know about the new PUB sockWn
until after it has already sent its message. (Excerpt included below)

However, in this design, where creating a new event handler is an extremely
frequent occurrence, and the code inside each event handler is very
lightweight and will only send one message, I am wondering what would be
the recommended solution?

Is there any way for sockWn to verify that sockA is connected to it? Unlike
many cases where a publisher should not know who is connected to it, in
this particular use case over inproc (not tcp), the PUB sockWn explicitly
expects to have XSUB sockA listening to it. That's why XSUB sockA did bind,
and PUB sockWn did connect.

I appreciate any suggestions.

Thanks,
Cosmo


[Excerpt from the guide]

There is one more important thing to know about PUB-SUB sockets: you do not
know precisely when a subscriber starts to get messages. Even if you start
a subscriber, wait a while, and then start the publisher, *the subscriber
will always miss the first messages that the publisher sends*. This is
because as the subscriber connects to the publisher (something that takes a
small but non-zero time), the publisher may already be sending messages out.

This "slow joiner" symptom hits enough people often enough that we're going
to explain it in detail. Remember that ØMQ does asynchronous I/O, i.e., in
the background. Say you have two nodes doing this, in this order:

   - Subscriber connects to an endpoint and receives and counts messages.
   - Publisher binds to an endpoint and immediately sends 1,000 messages.

Then the subscriber will most likely not receive anything. You'll blink,
check that you set a correct filter and try again, and the subscriber will
still not receive anything.

Making a TCP connection involves to and from handshaking that takes several
milliseconds depending on your network and the number of hops between
peers. In that time, ØMQ can send many messages. For sake of argument
assume it takes 5 msecs to establish a connection, and that same link can
handle 1M messages per second. During the 5 msecs that the subscriber is
connecting to the publisher, it takes the publisher only 1 msec to send out
those 1K messages.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.zeromq.org/pipermail/zeromq-dev/attachments/20140203/5b04926e/attachment.htm>


More information about the zeromq-dev mailing list