[zeromq-dev] memory pb

[Martin Sustrik]

Hi John,

> I am using ømq in a project (by the way I would like to thanks all of 
> you working on this product, I think that it is really great). I have a 
> problem with the pub/sub message queue. I have a publisher
> that is sending message at a rate of 10-20 MB/s. I have several consumer 
> but some of them can be slow. For slow consumer, the memory usage of the 
> publisher can be very high an finally it crash. I have tried to suse 
> high-watermark but I run into another problem. I am interesting in the 
> newest message not the oldest. Is there a way to make the publisher drop 
> the oldest message when the queue is full.
> I look the pub.cc and I found that when a pipe is full it goes into a 
> stalled mode. Can I force a pipe that is stalled to remove (decrease the 
> reference counter of a message) ? 
>  From what I've understand, ømq is using a lock free queue (only one 
> thread is reading and 1 is writing): is there a way to send a message or 
> a command that will for force the tail message to be read/removed ?

Dropping the oldest message is not implemented.

However, I would argue that that's not what you want anyway.

Imagine a message pipe between A and B. Some messages are queued in A's 
0MQ buffer, some are in A's TCP buffer, some are in swtch and router 
queues, some may be travelling through space (satellite connection) or 
ocean bottom. Some are in B's TCP buffer or B's 0MQ buffer.

Now, what 0MQ can possibly allow you to set is behavior when A's or B's 
message buffer is full.

So when A's (sender's) 0MQ buffer is full, oldest message in the buffer 
is dropped. However, there are older messages all over the downstream 
pipe. Thus, what happens is dropping some intermediate message rather 
than oldest one.

I'm not sure there's any real solution for the problem. What you can do 
is write your application in such a way that it drops messages if they 
seem to be too old.

Martin