[zeromq-dev] Forwarding ROUTER to PUB

Lindley French lindleyf at gmail.com
Thu Jan 16 22:19:01 CET 2014


I can see why. When I finally tracked down the place it happens in
dist.cpp, I realized it isn't as simple as changing a setting...the logic
just shrugs if the write fails. Hmm.


On Thu, Jan 16, 2014 at 4:04 PM, Charles Remes <lists at chuckremes.com> wrote:

> I don’t think there is any resistance to having pluggable HWM policies. No
> one has wanted this badly enough to write it.
>
> On Jan 16, 2014, at 2:38 PM, Lindley French <lindleyf at gmail.com> wrote:
>
> That will work, of course....I'm just curious what the resistance is to
> letting the HWM policy be settable.
>
>
> On Thu, Jan 16, 2014 at 3:27 PM, Goswin von Brederlow <goswin-v-b at web.de>wrote:
>
>> On Thu, Jan 16, 2014 at 02:45:31PM -0500, Lindley French wrote:
>> > > This is a common issue. If you can?t recover from dropped messages,
>> > PUB/SUB is not the correct pattern.
>> >
>> > In many cases, this is correct. I do not believe inproc is one of those
>> > cases, however. With inproc, you should have full control of who is
>> > subscribing and when they come up relative to the publisher. If your
>> > subscribers aren't running when you expect them to be running, that's a
>> > bug. If they aren't running fast enough, dropping messages *might* be a
>> > solution, or it might not. I don't feel that's a decision that can be
>> made
>> > in the general case.
>> >
>> > Let me put it this way: If I need one-to-many semantics with
>> backpressure
>> > and filtering, what should I use? PUB is the only one-to-many socket
>> type.
>> > I can write my own filtering code, keep a vector of push sockets, etc.
>> but
>> > that seems to defeat the point of ZMQ patterns. PUB is exactly what I
>> want
>> > in every way *except* the HWM behavior.
>> >
>> >
>> > On Thu, Jan 16, 2014 at 2:11 PM, Charles Remes <lists at chuckremes.com>
>> wrote:
>> >
>> > > This is a common issue. If you can?t recover from dropped messages,
>> > > PUB/SUB is not the correct pattern.
>> > >
>> > > This of PUB as a radio antenna. It broadcasts regardless of whether
>> or not
>> > > anyone is listening. If there are no listeners, every packet gets
>> dropped.
>> > > If a listener is slow, then packets will get dropped. If you need
>> further
>> > > guarantees about delivery, then you need to build some kind of
>> protocol
>> > > (ack, nak, ack window, etc) on top of DEALER/ROUTER.
>> > >
>> > > Also, as of libzmq 3.3, I believe the default HWM is 1000 (to prevent
>> > > memory exhaustion in the default configuration). If you want
>> ?infinite?
>> > > then setsockopt to -1.
>> > >
>> > > As for the dropped messages on inproc, you need to be careful to
>> confirm
>> > > that a listener (SUB) is actually up, running and *connected* before
>> you
>> > > start PUB?ing otherwise the PUB socket will drop messages.
>> Synchronization
>> > > for this is discussed in the guide. Alternately, just have your PUB
>> ?sleep?
>> > > for a second after the SUB bind/connects and you should be okay.
>> > >
>> > >
>> > > On Jan 16, 2014, at 1:03 PM, Lindley French <lindleyf at gmail.com>
>> wrote:
>> > >
>> > > Well, aside from the router issue I do like the arrangement for easily
>> > > handling different messages in different places. However, there may
>> be a
>> > > fatal flaw at the moment: PUB's desire to drop messages at the HWM.
>> While
>> > > making "drop" a default behavior for PUB is fine, I really don't like
>> that
>> > > it's the *only* behavior possible.
>> > >
>> > > Then again, that may or may not be the issue here. I haven't touched
>> the
>> > > HWM, so it should still be 0 which is theoretically infinite.
>> Nonetheless,
>> > > a bunch of my messages in a row vanished into the ether somewhere
>> between
>> > > PUB and SUB inproc sockets.
>> > >
>> > >
>> > > On Thu, Jan 16, 2014 at 1:02 PM, Lindley French <lindleyf at gmail.com
>> >wrote:
>> > >
>> > >> I tend to stuff in as many different features as I can when I'm first
>> > >> learning something new, it helps me get a feel for it.
>> > >>
>> > >> You should have seen my first major python program.....
>> > >>
>> > >>
>> > >> On Thu, Jan 16, 2014 at 12:46 PM, Charles Remes <
>> lists at chuckremes.com>wrote:
>> > >>
>> > >>> Create a socket for each worker thread and have your main thread
>> resend
>> > >>> the message down the appropriate socket. Sometimes it isn?t a good
>> idea to
>> > >>> try and shoe-horn every zeromq socket pattern into your app. :)
>> > >>>
>> > >>> On Jan 16, 2014, at 9:54 AM, Lindley French <lindleyf at gmail.com>
>> wrote:
>> > >>>
>> > >>> > A problem I was wrestling with was, how do I deal with a TCP
>> > >>> connection where messages of different types may arrive, and may
>> need to be
>> > >>> dealt with in different threads? The TCP socket can't be touched
>> directly
>> > >>> by multiple threads, of course. The obvious solution was to
>> immediately
>> > >>> forward messages arriving on the TCP socket to an inproc socket.
>> > >>> >
>> > >>> > I then took it one step further: why not make that inproc socket
>> a PUB
>> > >>> socket and make the first part of each message be a topic
>> identifier, so
>> > >>> that whichever thread knows how to deal with a particular message
>> can just
>> > >>> subscribe to it and ignore the rest?
>> > >>> >
>> > >>> > That's a great design, right up until I try to do it with the TCP
>> > >>> socket being a ROUTER. Now, no matter what the first part of the
>> sent
>> > >>> message is, the identity will end up being the first part on the
>> receiving
>> > >>> end. The PUB/SUB won't work without some tweaking.
>> > >>> >
>> > >>> > I don't want to just drop the identity; that's useful
>> information. I
>> > >>> could swap the first two parts; that will work, but it's
>> unintuitive and
>> > >>> could cause confusion down the road.
>> > >>> >
>> > >>> > Any other ideas?
>>
>> Add a splitter that simply checks the first frame, looks up the right
>> target socket and sends the remainder on the message onwards.
>> Then you can use a simple PUSH/PULL pattern.
>>
>> MfG
>>         Goswin
>> _______________________________________________
>> zeromq-dev mailing list
>> zeromq-dev at lists.zeromq.org
>> http://lists.zeromq.org/mailman/listinfo/zeromq-dev
>>
>
> _______________________________________________
> zeromq-dev mailing list
> zeromq-dev at lists.zeromq.org
> http://lists.zeromq.org/mailman/listinfo/zeromq-dev
>
>
>
> _______________________________________________
> zeromq-dev mailing list
> zeromq-dev at lists.zeromq.org
> http://lists.zeromq.org/mailman/listinfo/zeromq-dev
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.zeromq.org/pipermail/zeromq-dev/attachments/20140116/33694b7a/attachment.htm>


More information about the zeromq-dev mailing list