[zeromq-dev] Strategy for a Reverse HWM?

Martin Sustrik sustrik at 250bpm.com
Fri Feb 4 21:52:23 CET 2011


Hi Dave,

> The high water mark feature for pub-sub sockets is a nice way to prevent
> queue explosion. It works by throwing away new messages in favor
> existing messages on the queue. However, I am looking for the something
> like opposite - if a message has been hanging around unread, and new
> messages start coming in behind it, I want to throw away the old message
> in favor of the newer messages.

Throwing away the oldest message in impossible in distributed 
environment. Imagine the message passing through a sequence of 
intermediate nodes. When the ultimate publisher experiences HWM-exceeded 
condition, it should throw away the oldest message. The oldest message 
is most probably stored at the ultimate consumer N network hops away. 
Even worse, there may be a network congestion on the way, so even if you 
send a signal to the ultimate consumer to throw away the oldest message, 
it's not guaranteed the signal will make it to the destination in 
reasonable time. It can even make the congestion worse.

> Has anyone implemented anything like this? Ideally, there would even be
> a TTL, so that messages would get discarded after a certain amount of
> time, regardless of whether something else was coming in behind them.

Yes. TTL is a sane solution for the problem. It can be implemented in 
different ways. One way is to add a timestamp to each message and 
discard the expired messages. Another way would be to specify a message 
rate for the feed, allowing each node to discard next-to-send message if 
it is not able -- for whatever reason -- to keep up with the specified rate.

My gut feeling is that the latter solution is better as the former 
requires finding messages with expired TTL and dropping them even from 
the middle of the queue, which seems to be rather inefficient.

Anyway, nobody tried to implement that kind of thing yet.

Martin



More information about the zeromq-dev mailing list