[zeromq-dev] UDP Recv Buffer Sizes for OpenPGM/ZeroMQ

[Steven McCoy]

On 27 January 2011 17:24, Bob Beaty <drbobbeaty at gmail.com> wrote:

>   She is seeing the UDP receive buffers build to 128kB and then never
> empty. She is getting messages, but with the buffer full, she's concerned
> that she's dropping messages. While I haven't looked at the code, I'm
> guessing that OpenPGM will give me some exception if I have unrecoverable
> data loss, but please correct me if I'm wrong.
>   If you could set her (and therefore my) mind at ease with the explanation
> of why the receive buffer doesn't empty, I'd appreciate it. If it should
> empty, then I'll ask her to look at her hardware/OS setup as I don't believe
> we're incurring unrecoverable data loss.
>
>
It is expected with a high speed messaging layer that the buffers should
constantly be in use, typically for multicast one pushes the buffer size to
1MB and beyond to handle burst traffic.  Now OpenPGM itself is
single-threaded since version 2, this pushes the requirement for polling the
incoming data sockets, and minimizing buffer queuing, to ZeroMQ.  So from an
administration view you can monitor ZeroMQ's IO threads and determine
overhead.

If you encountered unrecoverable data loss then the PGM socket would reset
and ZeroMQ would subsequently attempt to re-open it after a delay which
would mitigate any ensuing broadcast storm.

The more pertinent issue is whether the data in the buffers are stale and
PGM is quietly enacting reliability, this tends to be more frequent on
Windows systems.  Typically on large deployments some form of monitoring
system is deployed to trace the network performance.  From a basic level you
can just run Wireshark or tcpdump and look for repair packets (RDATA).  With
OpenPGM I have SNMP & HTTP interfaces to monitor node performance but these
are unavailable in ZeroMQ unless you build it in yourself.  From a longer
term point of view there are many products available which perform
out-of-band monitoring specifically for such scenarios, slightly biased
recommendations of Rai Technologies Rai Insight and TS-Associates TipOff.

A preferable interim solution would be implementing some form of JMX/SNMP
style interface above ZeroMQ and so not having to rely on the older less
scalable protocols.  If you make such an interface it is pretty easy to pull
out all the MIB data from PGM.

-- 
Steve-o
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.zeromq.org/pipermail/zeromq-dev/attachments/20110128/5e75260c/attachment.htm>