[zeromq-dev] ZMQ_FLAT_PULL feature discussion

[Michael Haberler]

Am 13.08.2013 um 17:29 schrieb Daniel Krikun <krikun.daniel at gmail.com>:

> I have almost finished the feature, in the mean time, it evolved as follows:
> -- it will be available for ZMQ_PULL, ZMQ_SUB and ZMQ_DEALER sockets
> -- it is activated using ZMQ_CONFLATE socket option at the receiver side
> -- currently, multi-part messages would not be supported
> -- currently, would not be tested for (e)pgm
> -- receiver hwm is not considered
> 
> Does that makes sense to you?

not quite yet, but happy to read through commits

as I said, my key requirement is to have a publish always succeed even if it entails dropping old messages on the outbound queue; I dont fully understand if this is covered by your patch or not?

- Michael

> 
> I'm currently working on performance issues, hope to submit pull request soon :)
> 
> 
> On Tue, Aug 13, 2013 at 2:40 PM, Brian Knox <briank at talksum.com> wrote:
> I've been using ZeroMQ to distribute frames from video streams for processing and for some of my use cases this feature would be of great interest.  I'll happily be a test victim of a patch.
> 
> Brian
> 
> 
> On Tue, Aug 13, 2013 at 4:27 AM, Michael Haberler <mail17 at mah.priv.at> wrote:
> Daniel,
> 
> Am 07.06.2013 um 11:53 schrieb Daniel Krikun <krikun.daniel at gmail.com>:
> 
> > Hi all,
> >
> > I have a setup, where a server does graphics rendering based on client
> > requests, that is, clients send geometric data (position, orientation,
> > etc.) and the server runs in cycles: process incoming messages and do
> > some rendering.
> > Occasionally, the clients might be faster and few messages get
> > themselves queued on the server queue. However, only the most recent
> > message is of interest as the server does not renders the objects as
> > they were a couple of cycles before.
> >
> > To solve the problem, I have extended the ZMQ_ROUTER socket (via
> > subclassing) so that it has a background thread that empties the
> > socket message queue and stores the last message. The message is
> > stored in a double-buffer (one for each client, this is why I need a
> > router socket type). Then, when the server calls recv() on my extended
> > socket, it gets one of the stored messages (if there are any).
> >
> > I ask myself whether some similar functionality should be put inside
> > zeromq (say, under ZMQ_FLAT_PULL socket type), left as is in a "user
> > space" or maybe I should use UDP instead?
> 
> 
> having arrived at the point where I could make good use of the feature you proposed like so:
> 
> - my scenario happens in a PUB socket (not ROUTER); subscribers present but not necessarily constantly pulling messages
> - semantic summary is: 'the last message always wins' as it subsumes all the previous updates
> 
> the way I read the documentation on queue full/ high water mark behavior it looks like right now later queue adds are dropped and the old ones retained which is exactly the opposite what I'm looking for
> 
> I would think that kind of behavior could be implemented with a new socket option which twists the meaning of high water mark reached (let's call it ZMQ_SNDDROPOLD or so for now):
> 
> assuming this option is set and a queue add fails due to HWM reached, the sender would atomically pop entries from the queue head until the queue add succeeds
> 
> doesnt this match what you are proposing for ROUTER too? I dont see where a thread is needed here, but I'm not terribly read into ZMQ internals
> 
> curious about your progress!
> 
> - Michael
> 
> >
> >
> > Thank you,
> >
> > --
> > Daniel Krikun
> > _______________________________________________
> > zeromq-dev mailing list
> > zeromq-dev at lists.zeromq.org
> > http://lists.zeromq.org/mailman/listinfo/zeromq-dev
> 
> _______________________________________________
> zeromq-dev mailing list
> zeromq-dev at lists.zeromq.org
> http://lists.zeromq.org/mailman/listinfo/zeromq-dev
> 
> 
> _______________________________________________
> zeromq-dev mailing list
> zeromq-dev at lists.zeromq.org
> http://lists.zeromq.org/mailman/listinfo/zeromq-dev
> 
> 
> 
> 
> -- 
> Daniel Krikun
> _______________________________________________
> zeromq-dev mailing list
> zeromq-dev at lists.zeromq.org
> http://lists.zeromq.org/mailman/listinfo/zeromq-dev