[zeromq-dev] Why 0mq doesn't define TTL for message (for in-mem queue)?
Brian Knox
briank at talksum.com
Sat Dec 28 23:35:08 CET 2013
I disagree with the assertion that "ZeroMQ is a queuing solution", and I don't believe that ZeroMQ should try to be one. "ZeroMQ has and uses queues" != "ZeroMQ is a queuing solution", in my opinion.
From: zeromq-dev-bounces at lists.zeromq.org [mailto:zeromq-dev-bounces at lists.zeromq.org] On Behalf Of artemv zmq
Sent: Saturday, December 28, 2013 3:52 PM
To: ZeroMQ development list
Subject: Re: [zeromq-dev] Why 0mq doesn't define TTL for message (for in-mem queue)?
>> So if the traffic is not relevant, there is nothing IMHO to be done on ØMQ. Just add a TTL to a part of the multipart message, and have the receiver look at the time and discard the messages. Have an initial >> handshake to sync the peer timestamps just in case.
That's what I'm talking about ). Why this can't/shouldn't be done on 0mq? The thing is -- 0mq is queueing solution (after all) and TTL is part of any queueing. TTL is not concrete business feature, it's very common and ubiquitous thing.
Agree?
2013/12/28 Bruno D. Rodrigues <bruno.rodrigues at litux.org<mailto:bruno.rodrigues at litux.org>>
I recall apache activemq (jms) had a ttl system that worked like this (back in 2008, no idea how it works now):
- messages gets queued individually
- when a consumer connects, messages gets popped from the queue, and if the ttl elapsed, they'd get discarded instead of piped into the consumer.
this means that when no consumer was consuming, the TTL was irrelevant and messages would queue up anyway.
having a thread expiring these messages in parallel is quite hard to accomplish without some locks in the middle, which kills the performance.
Either way, the core problem, as far as I understood, is:
- messages configured with a TTL shall be discarded if the time elapse
The solution can include a second problem to solve:
- is the amount of traffic relevant? if so, it'd better be dropped on the sender, if not it can safely be dropped on the receiver.
So if the traffic is not relevant, there is nothing IMHO to be done on ØMQ. Just add a TTL to a part of the multipart message, and have the receiver look at the time and discard the messages. Have an initial handshake to sync the peer timestamps just in case.
If the traffic is relevant and the idea is to drop messages on the sender side, good luck implementing it. The simple "discard when piping out" should be easy to do, albeit I think ØMQ may already have a tcp buffer ready to send and it could be difficult to individualize the messages. The harder "discard always", as I said, it's quite hard to do without locks and killing performance.
On Dec 28, 2013, at 19:12, Brian Knox <briank at talksum.com<mailto:briank at talksum.com>> wrote:
I'm a little confused by the wording of your explanation. You say the "receiving peer" should pay attention to the "use-by date" attribute. If the peer is going to receive the message and check this attribute, I'm uncertain what a TTL that is checked by the actual zeromq queue is for in this use case.
From: zeromq-dev-bounces at lists.zeromq.org<mailto:zeromq-dev-bounces at lists.zeromq.org> [mailto:zeromq-dev-bounces at lists.zeromq.org] On Behalf Of artemv zmq
Sent: Saturday, December 28, 2013 1:34 PM
To: ZeroMQ development list
Subject: Re: [zeromq-dev] Why 0mq doesn't define TTL for message (for in-mem queue)?
> Perhaps. What problem would you be solving with TTLs, which is an
> issue today? ("Lack of feature X" is not a valid problem statement for
> feature X).
There's a need to send certain type of messages which have sort of "use-by date" attribute.
I.e. client may want to define this attribute and receiving peer should pay attention to it. What it would give? It would give a control over queued messages. Say, if I sending messages which
are "valid" _only_ when peers connected and "not valid" when peers aren't connected. Here "valid/not valid" is defined via "use-by date" message attribute. Receiving peer may check "use-by date" and recognize if gotten message was too long inside somebody's queue, and take some actions: log, throw error, silently discard a message, or even collect a message inside so called "dead letter" queue .
Btw, essentially, today 0mq defines "use-by date" as infinite. And proposition is to make this thing configurable.
2013/12/28 Pieter Hintjens <ph at imatix.com<mailto:ph at imatix.com>>
On Fri, Dec 27, 2013 at 1:26 PM, artemv zmq <artemv.zmq at gmail.com<mailto:artemv.zmq at gmail.com>> wrote:
> And all queueing solutions do have TTL .
Perhaps. What problem would you be solving with TTLs, which is an
issue today? ("Lack of feature X" is not a valid problem statement for
feature X).
-Pieter
_______________________________________________
zeromq-dev mailing list
zeromq-dev at lists.zeromq.org<mailto:zeromq-dev at lists.zeromq.org>
http://lists.zeromq.org/mailman/listinfo/zeromq-dev
_______________________________________________
zeromq-dev mailing list
zeromq-dev at lists.zeromq.org<mailto:zeromq-dev at lists.zeromq.org>
http://lists.zeromq.org/mailman/listinfo/zeromq-dev
_______________________________________________
zeromq-dev mailing list
zeromq-dev at lists.zeromq.org<mailto:zeromq-dev at lists.zeromq.org>
http://lists.zeromq.org/mailman/listinfo/zeromq-dev
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.zeromq.org/pipermail/zeromq-dev/attachments/20131228/8391d1bf/attachment.htm>
More information about the zeromq-dev
mailing list