[zeromq-dev] Discard old messages on XPUB before transmission

[CFK]

Thank you Peter, that was what I was afraid of.  Stephen, I agree with you
that having an option for dropping messages would be useful.  So, as a
strawman proposal, here are the options I'd like to see added to ZMQ:

ZMQ_EXPIRE_TIME_DELTA - If the time between when the user's call to send()
and the time the message is going to be sent is greater than this amount,
then the message is silently discarded.  This is useful for messages that
are useless if old.

ZMQ_PROBABILISTIC_HWM - Similar to the high water mark, except that it
isn't a hard limit.  When a message is up for transmission, ZMQ may choose
to drop the message.  The probability that a message will be dropped is
length(ZMQ output buffer)/ZMQ_PROBABILISTIC_HWM.  So the more messages
there are in the buffer, the greater the probability that the message at
the head of the queue will be dropped.  This allows back pressure to be
monitored by user applications.  For applications that can scale how much
data they transmit this can be VERY useful.  For example, I have several
options on how to compress aggregate/compress data.  Compression works
better on larger messages, but because I'm time sensitive, I want to get
messages out quickly.  By monitoring the back pressure, I can find the
sweet spot for my application, even as it changes.  Which brings up the
following options:

ZMQ_TRANSMISSION_SUCCESS_RATE_WINDOW and ZMQ_TRANSMISSION_SUCCESS_RATE -
ZMQ_TRANSMISSION_SUCCESS_RATE_WINDOW is an option on a socket.  ZMQ does a
moving average over the last ZMQ_TRANSMISSION_SUCCESS_RATE_WINDOW messages
that the user tried to send to calculate the fraction that were actually
transmitted (NOT received!  This is just on the sender's side, seeing what
ZMQ sent out).  You can monitor this via
zmq_socket_monitor<http://api.zeromq.org/3-2:zmq-socket-monitor>and
ZMQ_TRANSMISSION_SUCCESS_RATE.

Thoughts or comments?

Thanks,
Cem Karan



On Thu, Jul 18, 2013 at 1:44 PM, Pieter Hintjens <pieterh at gmail.com> wrote:

> There is no option to drop old messages.
>
> Pieter
> On Jul 18, 2013 4:45 PM, "CFK" <cfkaran2 at gmail.com> wrote:
>
>> Hi all, I'm trying to figure out what socket options I need to set under
>> ZMQ 3.2.3 to force ZMQ XPUB sockets to drop messages if they are old.  I
>> have time-sensitive messages, any/all of which can be dropped if they are
>> older than some amount (set by the option) before ZMQ starts actual
>> transmission. Is there an option for this?  From my testing, ZMQ_SNDTIMEO
>> does not seem to do this, and none of the other options I've read about
>> seem to be applicable. As it stands, I'm using pyzmq's MessageTracker
>> interface, along with a hackish thread interface, to queue all messages I
>> plan on sending until the currently sent message is actually transmitted,
>> before starting on the next message.  This is suboptimal, and I'd like to
>> do something better.
>>
>> Note that any solution MUST work with XPUB/XSUB as they operate over UDP
>> (via PGM).  I'm operating on a lossy wireless network, and cannot tolerate
>> how long TCP takes to timeout a connection, or how long it takes to figure
>> out a connection is back up.
>>
>> Thanks,
>> Cem Karan
>>
>> _______________________________________________
>> zeromq-dev mailing list
>> zeromq-dev at lists.zeromq.org
>> http://lists.zeromq.org/mailman/listinfo/zeromq-dev
>>
>>
> _______________________________________________
> zeromq-dev mailing list
> zeromq-dev at lists.zeromq.org
> http://lists.zeromq.org/mailman/listinfo/zeromq-dev
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.zeromq.org/pipermail/zeromq-dev/attachments/20130718/4c3b3867/attachment.htm>