[zeromq-dev] need to lower memory usage

Fri Jul 25 01:21:18 CEST 2014

Perhaps there's a way to run under valgrind or some other checker, and
see where that memory is being allocated? Working blind is not going
to be easy.

On Thu, Jul 24, 2014 at 7:23 PM, Philip Dizon <philipdotdev at gmail.com> wrote:
> Sorry to resurrect this old thread as I've been working on a more urgent
> project.
>
> Anyway, it seems that setting the max sockets has no effect on the amount of
> virtual memory used.  I tried setting max sockets to 1 and the memory still
> jumped up by 17MB after the first socket was created.
>
> Here's my sample code using just plain zmq and not czmq.
> void *acontext = zmq_ctx_new();
> zmq_ctx_set (acontext, ZMQ_MAX_SOCKETS, maxSockets);
> printf("ZMQ_MAX_SOCKETS = %d\n", zmq_ctx_get (acontext, ZMQ_MAX_SOCKETS));
> // 3MB virtual memory used at this point
> void *foo = zmq_socket (acontext, ZMQ_REP);  // 20MB virtual memory used at
> this point
>
>
> On Mon, Jun 23, 2014 at 10:44 PM, Pieter Hintjens <ph at imatix.com> wrote:
>>
>> In czmq master this is now configurable via zsys for the zsock API, which
>> doesn't use contexts.
>>
>> On Jun 24, 2014 6:55 AM, "Michel Pelletier" <pelletier.michel at gmail.com>
>> wrote:
>>>
>>> Yep, you can see a comment in the Python code I posted that demonstrates
>>> there is no underlying context until a socket is created.
>>>
>>> Why not create a dummy socket and the destroy it?  It's not pretty but it
>>> works.
>>>
>>> -Michel
>>>
>>>
>>> On Mon, Jun 23, 2014 at 9:16 PM, Steve Murphy <murf at parsetree.com> wrote:
>>>>
>>>> Michel--
>>>>
>>>> Just read a little deeper on zctx_underlying...
>>>>
>>>> The doc says:
>>>>
>>>> will be NULL before first socket
>>>> //  is created. Use with care.
>>>>
>>>> And, yet, in this use case, we would have to set
>>>> the MAX_SOCKETS before the first socket is created....
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> so, zctx_underlying() won't be useful.
>>>>
>>>>
>>>> murf
>>>>
>>>>
>>>>
>>>> On Mon, Jun 23, 2014 at 9:41 PM, Steve Murphy <murf at parsetree.com>
>>>> wrote:
>>>>>
>>>>> Thanks, Michel--
>>>>>
>>>>> missed that function in the zctx set!
>>>>> Many thanks!
>>>>>
>>>>> murf
>>>>>
>>>>>
>>>>>
>>>>> On Mon, Jun 23, 2014 at 8:56 PM, Michel Pelletier
>>>>> <pelletier.michel at gmail.com> wrote:
>>>>>>
>>>>>> You can use zctx_underlying to get the low level context object.  Here
>>>>>> is an example using pyczmq:
>>>>>>
>>>>>> >>> from pyczmq import zctx, zsocket, zmq
>>>>>> >>> s = zctx.new()
>>>>>> >>> zctx.underlying(s)  # None until a socket is made
>>>>>> >>> p = zsocket.new(s, zmq.PUSH)
>>>>>> >>> zmq.ctx_set(zctx.underlying(s), zmq.MAX_SOCKETS, 10)
>>>>>> 0
>>>>>> >>> zmq.ctx_get(zctx.underlying(s), zmq.MAX_SOCKETS)
>>>>>> 10
>>>>>> >>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Mon, Jun 23, 2014 at 7:26 PM, Steve Murphy <murf at parsetree.com>
>>>>>> wrote:
>>>>>>>
>>>>>>> Philip--
>>>>>>>
>>>>>>> In the code you provided, you are mixing czmq
>>>>>>> and zmq lib calls, and as stated, it won't work.
>>>>>>>
>>>>>>> The zctx_t that zctx_new provides is NOT a void*
>>>>>>> that zmq_ctx_new() would give you, and the calls to zmq_ctx_set() and
>>>>>>> .._get()
>>>>>>> will not work properly if given a zctx_t. If you replace your call
>>>>>>> to zctx_new() with zmq_ctx_new(), you will get better results.
>>>>>>>
>>>>>>> Now, Pieter mentioned that:
>>>>>>>
>>>>>>> "You can lower the max sockets per context, before creating your
>>>>>>> first
>>>>>>> context. See zmq_ctx_set ()."
>>>>>>>
>>>>>>> But I think he meant to say "before creating your first socket."
>>>>>>>
>>>>>>> (which, btw, is not in the ZMQ ref manual.)
>>>>>>>
>>>>>>> I double checked the zctx page in the CZMQ spec, and no function
>>>>>>> is available to get/set the context options... at least the
>>>>>>> MAX_SOCKETS, that is.
>>>>>>> So, if you need to play with MAX_SOCKETS, you have to abandon CZMQ,
>>>>>>> as there is no
>>>>>>> way to slip from the zctx_t world to the void* world.
>>>>>>>
>>>>>>> And a quick look at max sockets on a new context shows that the
>>>>>>> default for
>>>>>>> a new context is 1023. in the 4.1 stuff, there is also a
>>>>>>> ZMQ_SOCKET_LIMIT,
>>>>>>> which is the absolute maximum you can set in a set() call, but this
>>>>>>> isn't
>>>>>>> in the 4.0 versions.
>>>>>>>
>>>>>>> murf
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On Mon, Jun 23, 2014 at 10:25 AM, Philip Dizon
>>>>>>> <philipdotdev at gmail.com> wrote:
>>>>>>>>
>>>>>>>> Is it possible to set it ZMQ_MAX_SOCKETS for a czmq context object
>>>>>>>> because I'm getting an assert failure when I do a zmq_ctx_get?
>>>>>>>>
>>>>>>>>  e.g.
>>>>>>>>     client->ctx = zctx_new();
>>>>>>>>     assert(client->ctx);
>>>>>>>>
>>>>>>>>     int max_sockets = 256;
>>>>>>>>     zmq_ctx_set (client->ctx, ZMQ_MAX_SOCKETS, max_sockets);
>>>>>>>>     assert (zmq_ctx_get (client->ctx, ZMQ_MAX_SOCKETS) ==
>>>>>>>> max_sockets);
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> On Thu, Jun 19, 2014 at 4:05 PM, Pieter Hintjens <ph at imatix.com>
>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>> You can lower the max sockets per context, before creating your
>>>>>>>>> first
>>>>>>>>> context. See zmq_ctx_set ().
>>>>>>>>>
>>>>>>>>> On Fri, Jun 20, 2014 at 12:39 AM, Philip Dizon
>>>>>>>>> <philipdotdev at gmail.com> wrote:
>>>>>>>>> > Hi,
>>>>>>>>> >
>>>>>>>>> > I noticed that just creating a new zmq context and socket bumps
>>>>>>>>> > my memory
>>>>>>>>> > usage by 18MB, and this is a big problem on my embedded system
>>>>>>>>> > which only
>>>>>>>>> > contains about 100MB.  I determined this by using top command and
>>>>>>>>> > comparing
>>>>>>>>> > the difference in mem usage.
>>>>>>>>> > Is there any sort of option I can use to lower the memory usage?
>>>>>>>>> >
>>>>>>>>> > Thanks
>>>>>>>>> >
>>>>>>>>> > _______________________________________________
>>>>>>>>> > zeromq-dev mailing list
>>>>>>>>> > zeromq-dev at lists.zeromq.org
>>>>>>>>> > http://lists.zeromq.org/mailman/listinfo/zeromq-dev
>>>>>>>>> >
>>>>>>>>> _______________________________________________
>>>>>>>>> zeromq-dev mailing list
>>>>>>>>> zeromq-dev at lists.zeromq.org
>>>>>>>>> http://lists.zeromq.org/mailman/listinfo/zeromq-dev
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> _______________________________________________
>>>>>>>> zeromq-dev mailing list
>>>>>>>> zeromq-dev at lists.zeromq.org
>>>>>>>> http://lists.zeromq.org/mailman/listinfo/zeromq-dev
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>>
>>>>>>> Steve Murphy
>>>>>>> ParseTree Corporation
>>>>>>> 57 Lane 17
>>>>>>> Cody, WY 82414
>>>>>>> ✉  murf at parsetree dot com
>>>>>>> ☎ 307-899-5535
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> _______________________________________________
>>>>>>> zeromq-dev mailing list
>>>>>>> zeromq-dev at lists.zeromq.org
>>>>>>> http://lists.zeromq.org/mailman/listinfo/zeromq-dev
>>>>>>>
>>>>>>
>>>>>>
>>>>>> _______________________________________________
>>>>>> zeromq-dev mailing list
>>>>>> zeromq-dev at lists.zeromq.org
>>>>>> http://lists.zeromq.org/mailman/listinfo/zeromq-dev
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>>
>>>>> Steve Murphy
>>>>> ParseTree Corporation
>>>>> 57 Lane 17
>>>>> Cody, WY 82414
>>>>> ✉  murf at parsetree dot com
>>>>> ☎ 307-899-5535
>>>>>
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>>
>>>> Steve Murphy
>>>> ParseTree Corporation
>>>> 57 Lane 17
>>>> Cody, WY 82414
>>>> ✉  murf at parsetree dot com
>>>> ☎ 307-899-5535
>>>>
>>>>
>>>>
>>>> _______________________________________________
>>>> zeromq-dev mailing list
>>>> zeromq-dev at lists.zeromq.org
>>>> http://lists.zeromq.org/mailman/listinfo/zeromq-dev
>>>>
>>>
>>>
>>> _______________________________________________
>>> zeromq-dev mailing list
>>> zeromq-dev at lists.zeromq.org
>>> http://lists.zeromq.org/mailman/listinfo/zeromq-dev
>>>
>>
>> _______________________________________________
>> zeromq-dev mailing list
>> zeromq-dev at lists.zeromq.org
>> http://lists.zeromq.org/mailman/listinfo/zeromq-dev
>>
>
>
> _______________________________________________
> zeromq-dev mailing list
> zeromq-dev at lists.zeromq.org
> http://lists.zeromq.org/mailman/listinfo/zeromq-dev
>