[zeromq-dev] queue length

[Andrew Hume]

a queue length independent of TCP buffers etc would be fine.

maybe the problem is that i have an edge case.

in general, everything works well if you can use back-pressure from
processes farther down the pipeline to regulate the processing.
prcesses who always have a queue are the bottlenecks; those with no or
short queues are not the bottleneck.

unfortunately, the front end, or root process, for this flotilla of processes
is on the receiving end of a tcp socket that is delivering data at a rate
that can't be controlled. if this receiving process can't handle the data rate,
data gets discarded (at the sending side of this socket).

i had thought of using the queue to disk but not memory and somehow measuring that
but at my high data rate, i am scared of touching disk.

andrew

On Apr 21, 2011, at 10:31 PM, Martin Sustrik wrote:

> Hi Andrew,
> 
>> i know this was discussed earlier in the 2.0 context, but i can't recall
>> what the resolution was. is there a way to find out how many
>> messages are queued for a zmq socket? especially for PUSH/PULL.
>> i know its a hard problem in general, but i need something.
> 
> The problem is that messages can be stored in network buffers which we have no way of querying.
> 
>> my real problem is that i have a system of zeromq-connected processes
>> and the system runs out of memory. so far, its seems like zeromq is the
>> cause,
>> although it is no zeromq bug. it is because processes are queueing up
>> serious amounts of
>> messages thus enlarging the memory footprint.
>> if i had a notion of queue length (even a sloppy one), i could take
>> automatic action.
> 
> Why not set hard buffer limits (0MQ's HWMs & TCP buffer sizes)?
> 
> Martin


------------------
Andrew Hume  (best -> Telework) +1 623-551-2845
andrew at research.att.com  (Work) +1 973-236-2014
AT&T Labs - Research; member of USENIX and LOPSA




-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.zeromq.org/pipermail/zeromq-dev/attachments/20110422/7550c4ce/attachment.htm>