[zeromq-dev] queue length

[Matt Weinstein]

what transforms take place on the data?

is data being reduced, or can results be de-duplicated?

if so, what about using PGM? you could sequence # the data, use a hash to spread the data among k/n nodes, and de-duplicate after reduction.

On Apr 22, 2011, at 10:51 AM, Andrew Hume wrote:

> a queue length independent of TCP buffers etc would be fine.
> 
> maybe the problem is that i have an edge case.
> 
> in general, everything works well if you can use back-pressure from
> processes farther down the pipeline to regulate the processing.
> prcesses who always have a queue are the bottlenecks; those with no or
> short queues are not the bottleneck.
> 
> unfortunately, the front end, or root process, for this flotilla of processes
> is on the receiving end of a tcp socket that is delivering data at a rate
> that can't be controlled. if this receiving process can't handle the data rate,
> data gets discarded (at the sending side of this socket).
> 
> i had thought of using the queue to disk but not memory and somehow measuring that
> but at my high data rate, i am scared of touching disk.
> 
> andrew
> 
> On Apr 21, 2011, at 10:31 PM, Martin Sustrik wrote:
> 
>> Hi Andrew,
>> 
>>> i know this was discussed earlier in the 2.0 context, but i can't recall
>>> what the resolution was. is there a way to find out how many
>>> messages are queued for a zmq socket? especially for PUSH/PULL.
>>> i know its a hard problem in general, but i need something.
>> 
>> The problem is that messages can be stored in network buffers which we have no way of querying.
>> 
>>> my real problem is that i have a system of zeromq-connected processes
>>> and the system runs out of memory. so far, its seems like zeromq is the
>>> cause,
>>> although it is no zeromq bug. it is because processes are queueing up
>>> serious amounts of
>>> messages thus enlarging the memory footprint.
>>> if i had a notion of queue length (even a sloppy one), i could take
>>> automatic action.
>> 
>> Why not set hard buffer limits (0MQ's HWMs & TCP buffer sizes)?
>> 
>> Martin
> 
> 
> ------------------
> Andrew Hume  (best -> Telework) +1 623-551-2845
> andrew at research.att.com  (Work) +1 973-236-2014
> AT&T Labs - Research; member of USENIX and LOPSA
> 
> 
> 
> 
> _______________________________________________
> zeromq-dev mailing list
> zeromq-dev at lists.zeromq.org
> http://lists.zeromq.org/mailman/listinfo/zeromq-dev

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.zeromq.org/pipermail/zeromq-dev/attachments/20110422/058afb6f/attachment.htm>