[zeromq-dev] Why 0mq doesn't define TTL for message (for in-mem queue)?

Bruno D. Rodrigues bruno.rodrigues at litux.org
Sat Dec 28 20:30:59 CET 2013


I recall apache activemq (jms) had a ttl system that worked like this (back in 2008, no idea how it works now):

- messages gets queued individually
- when a consumer connects, messages gets popped from the queue, and if the ttl elapsed, they’d get discarded instead of piped into the consumer.

this means that when no consumer was consuming, the TTL was irrelevant and messages would queue up anyway.

having a thread expiring these messages in parallel is quite hard to accomplish without some locks in the middle, which kills the performance.


Either way, the core problem, as far as I understood, is:

- messages configured with a TTL shall be discarded if the time elapse

The solution can include a second problem to solve:
- is the amount of traffic relevant? if so, it’d better be dropped on the sender, if not it can safely be dropped on the receiver. 


So if the traffic is not relevant, there is nothing IMHO to be done on ØMQ. Just add a TTL to a part of the multipart message, and have the receiver look at the time and discard the messages. Have an initial handshake to sync the peer timestamps just in case.

If the traffic is relevant and the idea is to drop messages on the sender side, good luck implementing it. The simple “discard when piping out” should be easy to do, albeit I think ØMQ may already have a tcp buffer ready to send and it could be difficult to individualize the messages. The harder “discard always”, as I said, it’s quite hard to do without locks and killing performance.
 


On Dec 28, 2013, at 19:12, Brian Knox <briank at talksum.com> wrote:

> I’m a little confused by the wording of your explanation.  You say the “receiving peer” should pay attention to the “use-by date” attribute.  If the peer is going to receive the message and check this attribute, I’m uncertain what a TTL that is checked by the actual zeromq queue is for in this use case.
>  
> From: zeromq-dev-bounces at lists.zeromq.org [mailto:zeromq-dev-bounces at lists.zeromq.org] On Behalf Of artemv zmq
> Sent: Saturday, December 28, 2013 1:34 PM
> To: ZeroMQ development list
> Subject: Re: [zeromq-dev] Why 0mq doesn't define TTL for message (for in-mem queue)?
>  
> > Perhaps. What problem would you be solving with TTLs, which is an
> > issue today? ("Lack of feature X" is not a valid problem statement for
> > feature X).
>  
> There's a need to send certain type of messages which have sort of "use-by date" attribute. 
> I.e. client may want to define this attribute and receiving peer should pay attention to it.  What it would give?  It would give a control over queued messages. Say, if I sending messages which 
> are "valid" _only_ when peers connected  and  "not valid" when peers aren't connected.   Here "valid/not valid" is defined via "use-by date" message attribute. Receiving peer may check "use-by date" and recognize if gotten message was too long inside somebody's queue, and take some actions: log, throw error, silently discard a message, or even collect a message inside  so called "dead letter" queue .
>  
> Btw, essentially, today 0mq defines "use-by date" as infinite.  And proposition is to make this thing configurable.
>  
>  
>  
> 
> 2013/12/28 Pieter Hintjens <ph at imatix.com>
> On Fri, Dec 27, 2013 at 1:26 PM, artemv zmq <artemv.zmq at gmail.com> wrote:
> 
> > And all queueing solutions do have TTL .
> 
> Perhaps. What problem would you be solving with TTLs, which is an
> issue today? ("Lack of feature X" is not a valid problem statement for
> feature X).
> 
> -Pieter
> _______________________________________________
> zeromq-dev mailing list
> zeromq-dev at lists.zeromq.org
> http://lists.zeromq.org/mailman/listinfo/zeromq-dev
>  
> _______________________________________________
> zeromq-dev mailing list
> zeromq-dev at lists.zeromq.org
> http://lists.zeromq.org/mailman/listinfo/zeromq-dev

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.zeromq.org/pipermail/zeromq-dev/attachments/20131228/7cda1caa/attachment.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 235 bytes
Desc: Message signed with OpenPGP using GPGMail
URL: <https://lists.zeromq.org/pipermail/zeromq-dev/attachments/20131228/7cda1caa/attachment.sig>


More information about the zeromq-dev mailing list