[zeromq-dev] Slow consumer/subscriber, only get last message.

[Andrew Hume]

i do the same thing with one of my processes;
it gets about 20-100K inputs per second but takes 2-5s
to process one. so i have a reader thread, who interacts with the server main
like this:

main:
	// invariant: next_valid==1 means main owns it; ==0 means reader owns it
	for(;;){
		while(shared.next_valid == 0)
			nap(NAP_SECOND/10);
		process(shared.valid);
		shared.next_valid = 0;
	}

reader thread:
	for(;;){
		zmq_msg_init(&msg);
		zmq_recv(sock, &msg, 0);
		if(shared.next_valid == 0){
			strcpy(shared.valid, zmq_msg_data(&msg));
			shared.next_valid = 1;
		}
		zmq_msg_close(&msg);
	}

// for clarity, error checking elided above

safe and no mutexes.

On May 9, 2011, at 7:39 PM, Amr Ali wrote:

> If I were you I would put together a FIFO queue in a thread, receive early and
> often, put in memory, use a 0MQ inproc socket to another thread to _pull_ from
> that FIFO queue and process. 0MQ queuing should be used as a last resort, simply
> because you don't want to hog your infrastructure communication (that should be
> the case with most solutions).
> 
> So I'd advise that you free 0MQ from the burden early and often, however, if you
> don't care that much about delivery of every message (obviously), you could
> simply drop messages if 0MQ is filled up, but you also want to only show the
> last message (which is unpredictable to tell which is the last message), so you
> can't know which message is the last message, that brings us to the first
> solution, which is free 0MQ early and often by receiving all and processing later.
> 
> Hope that helps.
> 
> On 05/10/2011 04:12 AM, Beau Trepp wrote:
>> Howdy.
>> 
>> I'm using a PUB/SUB model to share information between a few processes (using
>> 0MQ of course). However some of the consumers are much slower than others. In
>> fact one is slower than the minimum publisher speed, which causes it to queue-up
>> lots of messages. I can't speed up the consumer at all, so I was hoping to have
>> it drop messages when it all becomes too much for it. It looks like by default
>> this will happen when I have hit the high water mark. However I get the
>> impression that this drops any new messages.
>> 
>> What I would like to happen is to drop all the messages in the consumer but the
>> last one. As this consumer only needs to show information to humans, it only
>> really needs the last message published. Of course to make it more difficult
>> this consumer is attached to multiple publishers. So what I want is for it to
>> keep the last message of each "filter" type for me to read, and the ones before
>> that I don't care about.
>> 
>> Any ideas?. I could introduce another thread to just read 0MQ messages, but it
>> seems like a waste.
>> 
>> Thanks,
>> Beau
>> 
>> 
>> 
>> _______________________________________________
>> zeromq-dev mailing list
>> zeromq-dev at lists.zeromq.org
>> http://lists.zeromq.org/mailman/listinfo/zeromq-dev
> 
> _______________________________________________
> zeromq-dev mailing list
> zeromq-dev at lists.zeromq.org
> http://lists.zeromq.org/mailman/listinfo/zeromq-dev


------------------
Andrew Hume  (best -> Telework) +1 623-551-2845
andrew at research.att.com  (Work) +1 973-236-2014
AT&T Labs - Research; member of USENIX and LOPSA




-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.zeromq.org/pipermail/zeromq-dev/attachments/20110509/4482820b/attachment.htm>