[zeromq-dev] ZMQ_FLAT_PULL feature discussion

[Daniel Krikun]

I have almost finished the feature, in the mean time, it evolved as follows:
-- it will be available for ZMQ_PULL, ZMQ_SUB and ZMQ_DEALER sockets
-- it is activated using ZMQ_CONFLATE socket option at the receiver side
-- currently, multi-part messages would not be supported
-- currently, would not be tested for (e)pgm
-- receiver hwm is not considered

Does that makes sense to you?

I'm currently working on performance issues, hope to submit pull request
soon :)


On Tue, Aug 13, 2013 at 2:40 PM, Brian Knox <briank at talksum.com> wrote:

> I've been using ZeroMQ to distribute frames from video streams for
> processing and for some of my use cases this feature would be of great
> interest.  I'll happily be a test victim of a patch.
>
> Brian
>
>
> On Tue, Aug 13, 2013 at 4:27 AM, Michael Haberler <mail17 at mah.priv.at>wrote:
>
>> Daniel,
>>
>> Am 07.06.2013 um 11:53 schrieb Daniel Krikun <krikun.daniel at gmail.com>:
>>
>> > Hi all,
>> >
>> > I have a setup, where a server does graphics rendering based on client
>> > requests, that is, clients send geometric data (position, orientation,
>> > etc.) and the server runs in cycles: process incoming messages and do
>> > some rendering.
>> > Occasionally, the clients might be faster and few messages get
>> > themselves queued on the server queue. However, only the most recent
>> > message is of interest as the server does not renders the objects as
>> > they were a couple of cycles before.
>> >
>> > To solve the problem, I have extended the ZMQ_ROUTER socket (via
>> > subclassing) so that it has a background thread that empties the
>> > socket message queue and stores the last message. The message is
>> > stored in a double-buffer (one for each client, this is why I need a
>> > router socket type). Then, when the server calls recv() on my extended
>> > socket, it gets one of the stored messages (if there are any).
>> >
>> > I ask myself whether some similar functionality should be put inside
>> > zeromq (say, under ZMQ_FLAT_PULL socket type), left as is in a "user
>> > space" or maybe I should use UDP instead?
>>
>>
>> having arrived at the point where I could make good use of the feature
>> you proposed like so:
>>
>> - my scenario happens in a PUB socket (not ROUTER); subscribers present
>> but not necessarily constantly pulling messages
>> - semantic summary is: 'the last message always wins' as it subsumes all
>> the previous updates
>>
>> the way I read the documentation on queue full/ high water mark behavior
>> it looks like right now later queue adds are dropped and the old ones
>> retained which is exactly the opposite what I'm looking for
>>
>> I would think that kind of behavior could be implemented with a new
>> socket option which twists the meaning of high water mark reached (let's
>> call it ZMQ_SNDDROPOLD or so for now):
>>
>> assuming this option is set and a queue add fails due to HWM reached, the
>> sender would atomically pop entries from the queue head until the queue add
>> succeeds
>>
>> doesnt this match what you are proposing for ROUTER too? I dont see where
>> a thread is needed here, but I'm not terribly read into ZMQ internals
>>
>> curious about your progress!
>>
>> - Michael
>>
>> >
>> >
>> > Thank you,
>> >
>> > --
>> > Daniel Krikun
>> > _______________________________________________
>> > zeromq-dev mailing list
>> > zeromq-dev at lists.zeromq.org
>> > http://lists.zeromq.org/mailman/listinfo/zeromq-dev
>>
>> _______________________________________________
>> zeromq-dev mailing list
>> zeromq-dev at lists.zeromq.org
>> http://lists.zeromq.org/mailman/listinfo/zeromq-dev
>>
>
>
> _______________________________________________
> zeromq-dev mailing list
> zeromq-dev at lists.zeromq.org
> http://lists.zeromq.org/mailman/listinfo/zeromq-dev
>
>


-- 
Daniel Krikun
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.zeromq.org/pipermail/zeromq-dev/attachments/20130813/d7578830/attachment.htm>