[zeromq-dev] Memory footprint of process bound to PUB socket increases as subscribers connect

Bill Torpey wallstprog at gmail.com
Fri Nov 6 15:55:46 CET 2020


Hi Harry:

Rather than “long standing issue” I would say “design limitation” — there’s an underlying assumption in ZeroMQ that sockets will get the chance to run process_commands on a regular basis, and if that doesn’t happen things can get “clogged up”.  I say design limitation because there’s really no “fix” — the best that can be done is what I would call a work-around, which is to make sure in the application/library that process_commands gets called from time to time.

Too bad that didn’t help, but perhaps the information is useful down the road.  Good luck.

Bill

> On Nov 5, 2020, at 7:36 PM, Harry Mander <harry.mander at taitradio.com> wrote:
> 
> Hi Bill,
> 
> Thanks for pointing me towards that - looking at the other issues linked in that report it seems this is a long standing issue with ZMQ... Unfortunately sending dummy messages, getting `ZMQ_EVENTS` with `zmq_getsockopt`, or calling `zmq_poll` on the PUB socket immediately after a message is sent still doesn't fix the issue even after SUB sockets are closed. I will keep digging... 
> 
> On Thu, 5 Nov 2020 at 02:56, Bill Torpey <wallstprog at gmail.com <mailto:wallstprog at gmail.com>> wrote:
> Hi Harry:
> 
> Not sure if this is relevant, but we had a similar-sounding problem — check out https://github.com/zeromq/libzmq/issues/3186 <https://github.com/zeromq/libzmq/issues/3186> and see if it might be helpful.
> 
> Regards,
> 
> Bill
> 
> 
>> On Nov 4, 2020, at 2:54 AM, Harry Mander via zeromq-dev <zeromq-dev at lists.zeromq.org <mailto:zeromq-dev at lists.zeromq.org>> wrote:
>> 
>> Hi,
>> 
>> I am having an unusual problem with memory usage and PUB/SUB sockets. Tested in C with libzmq 4.3.3 on Ubuntu 18.04.
>> 
>> I have a process which is bound to a PUB socket whose memory footprint (as measured by top, htop etc.) grows as other processes subscribe to the socket with SUB. However, the memory footprint of the PUB process does not increase when the SUB sockets are closed.
>> 
>> Publisher minimal test case, min-pub: https://pastebin.com/r85GZYcw <https://pastebin.com/r85GZYcw>
>> Subscriber minimal test case, min-sub: https://pastebin.com/9cw7tNgL <https://pastebin.com/9cw7tNgL>
>> 
>> min-pub polls stdin for messages to send on `ipc://@test` <> socket. Memory footprint increase can be seen on my machine by running `min-sub 100 info` which creates 100 subscribers connected to `ipc://@test` <> and subscribes each one to "info". Typing "info message" into min-pub causes messages to be received fine in min-sub process. min-sub is closed by sending SIGINT (i.e. Ctrl-C) before which the SUB sockets will unsubscribe from "info" and be closed. min-pub is closed by sending EOF (i.e. Ctrl-D).
>> 
>> Each time min-sub is run the resident memory footprint of min-pub as observed in htop increases and never decreases even after min-sub exits. If min-sub is run enough times min-pub will be terminated by the OOM killer.
>> 
>> I have tested both sub and pub programs with memcheck and neither reports any errors or memory leaks.
>> 
>> Regards,
>> Harry
>> 
>> This Communication is Confidential. We only send and receive email on the
>> basis of the terms set out at www.taitradio.com/email_disclaimer <http://www.taitradio.com/email_disclaimer>
>> _______________________________________________
>> zeromq-dev mailing list
>> zeromq-dev at lists.zeromq.org <mailto:zeromq-dev at lists.zeromq.org>
>> https://lists.zeromq.org/mailman/listinfo/zeromq-dev <https://lists.zeromq.org/mailman/listinfo/zeromq-dev>
> 
> 
> 
> -- 
> Harry Mander
> Junior Design Engineer
> Tait Communications
> Email: harry.mander at taitradio.com <mailto:harry.mander at taitradio.com>
> 
> 
> www.taitradio.com <http://www.taitradio.com/>
> 
> This Communication is Confidential. We only send and receive email on the
> basis of the terms set out at www.taitradio.com/email_disclaimer <http://www.taitradio.com/email_disclaimer>

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.zeromq.org/pipermail/zeromq-dev/attachments/20201106/b9a9c75c/attachment.htm>


More information about the zeromq-dev mailing list