[zeromq-dev] Forwarding ROUTER to PUB
Lindley French
lindleyf at gmail.com
Thu Jan 16 20:45:31 CET 2014
> This is a common issue. If you can’t recover from dropped messages,
PUB/SUB is not the correct pattern.
In many cases, this is correct. I do not believe inproc is one of those
cases, however. With inproc, you should have full control of who is
subscribing and when they come up relative to the publisher. If your
subscribers aren't running when you expect them to be running, that's a
bug. If they aren't running fast enough, dropping messages *might* be a
solution, or it might not. I don't feel that's a decision that can be made
in the general case.
Let me put it this way: If I need one-to-many semantics with backpressure
and filtering, what should I use? PUB is the only one-to-many socket type.
I can write my own filtering code, keep a vector of push sockets, etc. but
that seems to defeat the point of ZMQ patterns. PUB is exactly what I want
in every way *except* the HWM behavior.
On Thu, Jan 16, 2014 at 2:11 PM, Charles Remes <lists at chuckremes.com> wrote:
> This is a common issue. If you can’t recover from dropped messages,
> PUB/SUB is not the correct pattern.
>
> This of PUB as a radio antenna. It broadcasts regardless of whether or not
> anyone is listening. If there are no listeners, every packet gets dropped.
> If a listener is slow, then packets will get dropped. If you need further
> guarantees about delivery, then you need to build some kind of protocol
> (ack, nak, ack window, etc) on top of DEALER/ROUTER.
>
> Also, as of libzmq 3.3, I believe the default HWM is 1000 (to prevent
> memory exhaustion in the default configuration). If you want “infinite”
> then setsockopt to -1.
>
> As for the dropped messages on inproc, you need to be careful to confirm
> that a listener (SUB) is actually up, running and *connected* before you
> start PUB’ing otherwise the PUB socket will drop messages. Synchronization
> for this is discussed in the guide. Alternately, just have your PUB “sleep”
> for a second after the SUB bind/connects and you should be okay.
>
>
> On Jan 16, 2014, at 1:03 PM, Lindley French <lindleyf at gmail.com> wrote:
>
> Well, aside from the router issue I do like the arrangement for easily
> handling different messages in different places. However, there may be a
> fatal flaw at the moment: PUB's desire to drop messages at the HWM. While
> making "drop" a default behavior for PUB is fine, I really don't like that
> it's the *only* behavior possible.
>
> Then again, that may or may not be the issue here. I haven't touched the
> HWM, so it should still be 0 which is theoretically infinite. Nonetheless,
> a bunch of my messages in a row vanished into the ether somewhere between
> PUB and SUB inproc sockets.
>
>
> On Thu, Jan 16, 2014 at 1:02 PM, Lindley French <lindleyf at gmail.com>wrote:
>
>> I tend to stuff in as many different features as I can when I'm first
>> learning something new, it helps me get a feel for it.
>>
>> You should have seen my first major python program.....
>>
>>
>> On Thu, Jan 16, 2014 at 12:46 PM, Charles Remes <lists at chuckremes.com>wrote:
>>
>>> Create a socket for each worker thread and have your main thread resend
>>> the message down the appropriate socket. Sometimes it isn’t a good idea to
>>> try and shoe-horn every zeromq socket pattern into your app. :)
>>>
>>> On Jan 16, 2014, at 9:54 AM, Lindley French <lindleyf at gmail.com> wrote:
>>>
>>> > A problem I was wrestling with was, how do I deal with a TCP
>>> connection where messages of different types may arrive, and may need to be
>>> dealt with in different threads? The TCP socket can't be touched directly
>>> by multiple threads, of course. The obvious solution was to immediately
>>> forward messages arriving on the TCP socket to an inproc socket.
>>> >
>>> > I then took it one step further: why not make that inproc socket a PUB
>>> socket and make the first part of each message be a topic identifier, so
>>> that whichever thread knows how to deal with a particular message can just
>>> subscribe to it and ignore the rest?
>>> >
>>> > That's a great design, right up until I try to do it with the TCP
>>> socket being a ROUTER. Now, no matter what the first part of the sent
>>> message is, the identity will end up being the first part on the receiving
>>> end. The PUB/SUB won't work without some tweaking.
>>> >
>>> > I don't want to just drop the identity; that's useful information. I
>>> could swap the first two parts; that will work, but it's unintuitive and
>>> could cause confusion down the road.
>>> >
>>> > Any other ideas?
>>> > _______________________________________________
>>> > zeromq-dev mailing list
>>> > zeromq-dev at lists.zeromq.org
>>> > http://lists.zeromq.org/mailman/listinfo/zeromq-dev
>>>
>>> _______________________________________________
>>> zeromq-dev mailing list
>>> zeromq-dev at lists.zeromq.org
>>> http://lists.zeromq.org/mailman/listinfo/zeromq-dev
>>>
>>
>>
> _______________________________________________
> zeromq-dev mailing list
> zeromq-dev at lists.zeromq.org
> http://lists.zeromq.org/mailman/listinfo/zeromq-dev
>
>
>
> _______________________________________________
> zeromq-dev mailing list
> zeromq-dev at lists.zeromq.org
> http://lists.zeromq.org/mailman/listinfo/zeromq-dev
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.zeromq.org/pipermail/zeromq-dev/attachments/20140116/27da9601/attachment.htm>
More information about the zeromq-dev
mailing list