[zeromq-dev] ZMQ_FLAT_PULL feature discussion

[Daniel Krikun]

I've submitted a pull-request for the feature:
https://github.com/zeromq/libzmq/pull/626

Best,


On Wed, Aug 14, 2013 at 10:37 PM, Daniel Krikun <krikun.daniel at gmail.com>wrote:

> I see the point, so I will allow setting the option also on the sender
> side.
>
>
> On Wed, Aug 14, 2013 at 1:07 PM, Michael Haberler <mail17 at mah.priv.at>wrote:
>
>>
>> Am 13.08.2013 um 19:23 schrieb Daniel Krikun <krikun.daniel at gmail.com>:
>>
>> > For the sender, the situation is transparent, however, it so happens
>> that for ZMQ_PUB if hwm is reached it indeed drops the message so it should
>> work for you, right?
>>
>> my requirement is that PUB drops the oldest messages, not the last one,
>> of the hwm is reached - is that what you suggest works?
>>
>> -m
>>
>> >
>> >
>> > On Tue, Aug 13, 2013 at 6:59 PM, Michael Haberler <mail17 at mah.priv.at>
>> wrote:
>> >
>> > Am 13.08.2013 um 17:29 schrieb Daniel Krikun <krikun.daniel at gmail.com>:
>> >
>> > > I have almost finished the feature, in the mean time, it evolved as
>> follows:
>> > > -- it will be available for ZMQ_PULL, ZMQ_SUB and ZMQ_DEALER sockets
>> > > -- it is activated using ZMQ_CONFLATE socket option at the receiver
>> side
>> > > -- currently, multi-part messages would not be supported
>> > > -- currently, would not be tested for (e)pgm
>> > > -- receiver hwm is not considered
>> > >
>> > > Does that makes sense to you?
>> >
>> > not quite yet, but happy to read through commits
>> >
>> > as I said, my key requirement is to have a publish always succeed even
>> if it entails dropping old messages on the outbound queue; I dont fully
>> understand if this is covered by your patch or not?
>> >
>> > - Michael
>> >
>> > >
>> > > I'm currently working on performance issues, hope to submit pull
>> request soon :)
>> > >
>> > >
>> > > On Tue, Aug 13, 2013 at 2:40 PM, Brian Knox <briank at talksum.com>
>> wrote:
>> > > I've been using ZeroMQ to distribute frames from video streams for
>> processing and for some of my use cases this feature would be of great
>> interest.  I'll happily be a test victim of a patch.
>> > >
>> > > Brian
>> > >
>> > >
>> > > On Tue, Aug 13, 2013 at 4:27 AM, Michael Haberler <mail17 at mah.priv.at>
>> wrote:
>> > > Daniel,
>> > >
>> > > Am 07.06.2013 um 11:53 schrieb Daniel Krikun <krikun.daniel at gmail.com
>> >:
>> > >
>> > > > Hi all,
>> > > >
>> > > > I have a setup, where a server does graphics rendering based on
>> client
>> > > > requests, that is, clients send geometric data (position,
>> orientation,
>> > > > etc.) and the server runs in cycles: process incoming messages and
>> do
>> > > > some rendering.
>> > > > Occasionally, the clients might be faster and few messages get
>> > > > themselves queued on the server queue. However, only the most recent
>> > > > message is of interest as the server does not renders the objects as
>> > > > they were a couple of cycles before.
>> > > >
>> > > > To solve the problem, I have extended the ZMQ_ROUTER socket (via
>> > > > subclassing) so that it has a background thread that empties the
>> > > > socket message queue and stores the last message. The message is
>> > > > stored in a double-buffer (one for each client, this is why I need a
>> > > > router socket type). Then, when the server calls recv() on my
>> extended
>> > > > socket, it gets one of the stored messages (if there are any).
>> > > >
>> > > > I ask myself whether some similar functionality should be put inside
>> > > > zeromq (say, under ZMQ_FLAT_PULL socket type), left as is in a "user
>> > > > space" or maybe I should use UDP instead?
>> > >
>> > >
>> > > having arrived at the point where I could make good use of the
>> feature you proposed like so:
>> > >
>> > > - my scenario happens in a PUB socket (not ROUTER); subscribers
>> present but not necessarily constantly pulling messages
>> > > - semantic summary is: 'the last message always wins' as it subsumes
>> all the previous updates
>> > >
>> > > the way I read the documentation on queue full/ high water mark
>> behavior it looks like right now later queue adds are dropped and the old
>> ones retained which is exactly the opposite what I'm looking for
>> > >
>> > > I would think that kind of behavior could be implemented with a new
>> socket option which twists the meaning of high water mark reached (let's
>> call it ZMQ_SNDDROPOLD or so for now):
>> > >
>> > > assuming this option is set and a queue add fails due to HWM reached,
>> the sender would atomically pop entries from the queue head until the queue
>> add succeeds
>> > >
>> > > doesnt this match what you are proposing for ROUTER too? I dont see
>> where a thread is needed here, but I'm not terribly read into ZMQ internals
>> > >
>> > > curious about your progress!
>> > >
>> > > - Michael
>> > >
>> > > >
>> > > >
>> > > > Thank you,
>> > > >
>> > > > --
>> > > > Daniel Krikun
>> > > > _______________________________________________
>> > > > zeromq-dev mailing list
>> > > > zeromq-dev at lists.zeromq.org
>> > > > http://lists.zeromq.org/mailman/listinfo/zeromq-dev
>> > >
>> > > _______________________________________________
>> > > zeromq-dev mailing list
>> > > zeromq-dev at lists.zeromq.org
>> > > http://lists.zeromq.org/mailman/listinfo/zeromq-dev
>> > >
>> > >
>> > > _______________________________________________
>> > > zeromq-dev mailing list
>> > > zeromq-dev at lists.zeromq.org
>> > > http://lists.zeromq.org/mailman/listinfo/zeromq-dev
>> > >
>> > >
>> > >
>> > >
>> > > --
>> > > Daniel Krikun
>> > > _______________________________________________
>> > > zeromq-dev mailing list
>> > > zeromq-dev at lists.zeromq.org
>> > > http://lists.zeromq.org/mailman/listinfo/zeromq-dev
>> >
>> > _______________________________________________
>> > zeromq-dev mailing list
>> > zeromq-dev at lists.zeromq.org
>> > http://lists.zeromq.org/mailman/listinfo/zeromq-dev
>> >
>> >
>> >
>> > --
>> > Daniel Krikun
>> > _______________________________________________
>> > zeromq-dev mailing list
>> > zeromq-dev at lists.zeromq.org
>> > http://lists.zeromq.org/mailman/listinfo/zeromq-dev
>>
>> _______________________________________________
>> zeromq-dev mailing list
>> zeromq-dev at lists.zeromq.org
>> http://lists.zeromq.org/mailman/listinfo/zeromq-dev
>>
>
>
>
> --
> Daniel Krikun
>



-- 
Daniel Krikun
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.zeromq.org/pipermail/zeromq-dev/attachments/20130818/99a0362c/attachment.htm>