[zeromq-dev] Forwarding ROUTER to PUB

Goswin von Brederlow goswin-v-b at web.de
Thu Jan 16 21:27:33 CET 2014

On Thu, Jan 16, 2014 at 02:45:31PM -0500, Lindley French wrote:
> > This is a common issue. If you can?t recover from dropped messages,
> PUB/SUB is not the correct pattern.
> In many cases, this is correct. I do not believe inproc is one of those
> cases, however. With inproc, you should have full control of who is
> subscribing and when they come up relative to the publisher. If your
> subscribers aren't running when you expect them to be running, that's a
> bug. If they aren't running fast enough, dropping messages *might* be a
> solution, or it might not. I don't feel that's a decision that can be made
> in the general case.
> Let me put it this way: If I need one-to-many semantics with backpressure
> and filtering, what should I use? PUB is the only one-to-many socket type.
> I can write my own filtering code, keep a vector of push sockets, etc. but
> that seems to defeat the point of ZMQ patterns. PUB is exactly what I want
> in every way *except* the HWM behavior.
> On Thu, Jan 16, 2014 at 2:11 PM, Charles Remes <lists at chuckremes.com> wrote:
> > This is a common issue. If you can?t recover from dropped messages,
> > PUB/SUB is not the correct pattern.
> >
> > This of PUB as a radio antenna. It broadcasts regardless of whether or not
> > anyone is listening. If there are no listeners, every packet gets dropped.
> > If a listener is slow, then packets will get dropped. If you need further
> > guarantees about delivery, then you need to build some kind of protocol
> > (ack, nak, ack window, etc) on top of DEALER/ROUTER.
> >
> > Also, as of libzmq 3.3, I believe the default HWM is 1000 (to prevent
> > memory exhaustion in the default configuration). If you want ?infinite?
> > then setsockopt to -1.
> >
> > As for the dropped messages on inproc, you need to be careful to confirm
> > that a listener (SUB) is actually up, running and *connected* before you
> > start PUB?ing otherwise the PUB socket will drop messages. Synchronization
> > for this is discussed in the guide. Alternately, just have your PUB ?sleep?
> > for a second after the SUB bind/connects and you should be okay.
> >
> >
> > On Jan 16, 2014, at 1:03 PM, Lindley French <lindleyf at gmail.com> wrote:
> >
> > Well, aside from the router issue I do like the arrangement for easily
> > handling different messages in different places. However, there may be a
> > fatal flaw at the moment: PUB's desire to drop messages at the HWM. While
> > making "drop" a default behavior for PUB is fine, I really don't like that
> > it's the *only* behavior possible.
> >
> > Then again, that may or may not be the issue here. I haven't touched the
> > HWM, so it should still be 0 which is theoretically infinite. Nonetheless,
> > a bunch of my messages in a row vanished into the ether somewhere between
> > PUB and SUB inproc sockets.
> >
> >
> > On Thu, Jan 16, 2014 at 1:02 PM, Lindley French <lindleyf at gmail.com>wrote:
> >
> >> I tend to stuff in as many different features as I can when I'm first
> >> learning something new, it helps me get a feel for it.
> >>
> >> You should have seen my first major python program.....
> >>
> >>
> >> On Thu, Jan 16, 2014 at 12:46 PM, Charles Remes <lists at chuckremes.com>wrote:
> >>
> >>> Create a socket for each worker thread and have your main thread resend
> >>> the message down the appropriate socket. Sometimes it isn?t a good idea to
> >>> try and shoe-horn every zeromq socket pattern into your app. :)
> >>>
> >>> On Jan 16, 2014, at 9:54 AM, Lindley French <lindleyf at gmail.com> wrote:
> >>>
> >>> > A problem I was wrestling with was, how do I deal with a TCP
> >>> connection where messages of different types may arrive, and may need to be
> >>> dealt with in different threads? The TCP socket can't be touched directly
> >>> by multiple threads, of course. The obvious solution was to immediately
> >>> forward messages arriving on the TCP socket to an inproc socket.
> >>> >
> >>> > I then took it one step further: why not make that inproc socket a PUB
> >>> socket and make the first part of each message be a topic identifier, so
> >>> that whichever thread knows how to deal with a particular message can just
> >>> subscribe to it and ignore the rest?
> >>> >
> >>> > That's a great design, right up until I try to do it with the TCP
> >>> socket being a ROUTER. Now, no matter what the first part of the sent
> >>> message is, the identity will end up being the first part on the receiving
> >>> end. The PUB/SUB won't work without some tweaking.
> >>> >
> >>> > I don't want to just drop the identity; that's useful information. I
> >>> could swap the first two parts; that will work, but it's unintuitive and
> >>> could cause confusion down the road.
> >>> >
> >>> > Any other ideas?

Add a splitter that simply checks the first frame, looks up the right
target socket and sends the remainder on the message onwards.
Then you can use a simple PUSH/PULL pattern.


More information about the zeromq-dev mailing list