[zeromq-dev] Discard old messages on XPUB before transmission

Diego Duclos diego.duclos at gmail.com
Thu Jul 18 22:12:26 CEST 2013


I would have use for some of this stuff as well.
However, I don't think it really feels at home in zmq itself ? What about a
middleware layer between zmq and user apps that takes care of this ?

Kr,

Diego Duclos


On Thu, Jul 18, 2013 at 8:47 PM, CFK <cfkaran2 at gmail.com> wrote:

> Thank you Peter, that was what I was afraid of.  Stephen, I agree with you
> that having an option for dropping messages would be useful.  So, as a
> strawman proposal, here are the options I'd like to see added to ZMQ:
>
> ZMQ_EXPIRE_TIME_DELTA - If the time between when the user's call to send()
> and the time the message is going to be sent is greater than this amount,
> then the message is silently discarded.  This is useful for messages that
> are useless if old.
>
> ZMQ_PROBABILISTIC_HWM - Similar to the high water mark, except that it
> isn't a hard limit.  When a message is up for transmission, ZMQ may choose
> to drop the message.  The probability that a message will be dropped is
> length(ZMQ output buffer)/ZMQ_PROBABILISTIC_HWM.  So the more messages
> there are in the buffer, the greater the probability that the message at
> the head of the queue will be dropped.  This allows back pressure to be
> monitored by user applications.  For applications that can scale how much
> data they transmit this can be VERY useful.  For example, I have several
> options on how to compress aggregate/compress data.  Compression works
> better on larger messages, but because I'm time sensitive, I want to get
> messages out quickly.  By monitoring the back pressure, I can find the
> sweet spot for my application, even as it changes.  Which brings up the
> following options:
>
> ZMQ_TRANSMISSION_SUCCESS_RATE_WINDOW and ZMQ_TRANSMISSION_SUCCESS_RATE -
> ZMQ_TRANSMISSION_SUCCESS_RATE_WINDOW is an option on a socket.  ZMQ does a
> moving average over the last ZMQ_TRANSMISSION_SUCCESS_RATE_WINDOW messages
> that the user tried to send to calculate the fraction that were actually
> transmitted (NOT received!  This is just on the sender's side, seeing what
> ZMQ sent out).  You can monitor this via zmq_socket_monitor<http://api.zeromq.org/3-2:zmq-socket-monitor>and ZMQ_TRANSMISSION_SUCCESS_RATE.
>
> Thoughts or comments?
>
> Thanks,
> Cem Karan
>
>
>
> On Thu, Jul 18, 2013 at 1:44 PM, Pieter Hintjens <pieterh at gmail.com>wrote:
>
>> There is no option to drop old messages.
>>
>> Pieter
>> On Jul 18, 2013 4:45 PM, "CFK" <cfkaran2 at gmail.com> wrote:
>>
>>> Hi all, I'm trying to figure out what socket options I need to set under
>>> ZMQ 3.2.3 to force ZMQ XPUB sockets to drop messages if they are old.  I
>>> have time-sensitive messages, any/all of which can be dropped if they are
>>> older than some amount (set by the option) before ZMQ starts actual
>>> transmission. Is there an option for this?  From my testing, ZMQ_SNDTIMEO
>>> does not seem to do this, and none of the other options I've read about
>>> seem to be applicable. As it stands, I'm using pyzmq's MessageTracker
>>> interface, along with a hackish thread interface, to queue all messages I
>>> plan on sending until the currently sent message is actually transmitted,
>>> before starting on the next message.  This is suboptimal, and I'd like to
>>> do something better.
>>>
>>> Note that any solution MUST work with XPUB/XSUB as they operate over UDP
>>> (via PGM).  I'm operating on a lossy wireless network, and cannot tolerate
>>> how long TCP takes to timeout a connection, or how long it takes to figure
>>> out a connection is back up.
>>>
>>> Thanks,
>>> Cem Karan
>>>
>>> _______________________________________________
>>> zeromq-dev mailing list
>>> zeromq-dev at lists.zeromq.org
>>> http://lists.zeromq.org/mailman/listinfo/zeromq-dev
>>>
>>>
>> _______________________________________________
>> zeromq-dev mailing list
>> zeromq-dev at lists.zeromq.org
>> http://lists.zeromq.org/mailman/listinfo/zeromq-dev
>>
>>
>
> _______________________________________________
> zeromq-dev mailing list
> zeromq-dev at lists.zeromq.org
> http://lists.zeromq.org/mailman/listinfo/zeromq-dev
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.zeromq.org/pipermail/zeromq-dev/attachments/20130718/2fe50e03/attachment.htm>


More information about the zeromq-dev mailing list