[zeromq-dev] need to lower memory usage

Thu Jul 24 19:23:04 CEST 2014

Sorry to resurrect this old thread as I've been working on a more urgent
project.

Anyway, it seems that setting the max sockets has no effect on the amount
of virtual memory used.  I tried setting max sockets to 1 and the memory
still jumped up by 17MB after the first socket was created.

Here's my sample code using just plain zmq and not czmq.
void *acontext = zmq_ctx_new();
zmq_ctx_set (acontext, ZMQ_MAX_SOCKETS, maxSockets);
printf("ZMQ_MAX_SOCKETS = %d\n", zmq_ctx_get (acontext, ZMQ_MAX_SOCKETS));
// 3MB virtual memory used at this point
void *foo = zmq_socket (acontext, ZMQ_REP);  // 20MB virtual memory used at
this point

On Mon, Jun 23, 2014 at 10:44 PM, Pieter Hintjens <ph at imatix.com> wrote:

> In czmq master this is now configurable via zsys for the zsock API, which
> doesn't use contexts.
> On Jun 24, 2014 6:55 AM, "Michel Pelletier" <pelletier.michel at gmail.com>
> wrote:
>
>> Yep, you can see a comment in the Python code I posted that demonstrates
>> there is no underlying context until a socket is created.
>>
>> Why not create a dummy socket and the destroy it?  It's not pretty but it
>> works.
>>
>> -Michel
>>
>>
>> On Mon, Jun 23, 2014 at 9:16 PM, Steve Murphy <murf at parsetree.com> wrote:
>>
>>> Michel--
>>>
>>> Just read a little deeper on zctx_underlying...
>>>
>>> The doc says:
>>>
>>> will be NULL before first socket
>>> //  is created. Use with care.
>>>
>>> And, yet, in this use case, we would have to set
>>> the MAX_SOCKETS before the first socket is created....
>>>
>>>
>>>
>>> so, zctx_underlying() won't be useful.
>>>
>>>
>>> murf
>>>
>>>
>>>
>>> On Mon, Jun 23, 2014 at 9:41 PM, Steve Murphy <murf at parsetree.com>
>>> wrote:
>>>
>>>> Thanks, Michel--
>>>>
>>>> missed that function in the zctx set!
>>>> Many thanks!
>>>>
>>>> murf
>>>>
>>>>
>>>>
>>>> On Mon, Jun 23, 2014 at 8:56 PM, Michel Pelletier <
>>>> pelletier.michel at gmail.com> wrote:
>>>>
>>>>> You can use zctx_underlying to get the low level context object.  Here
>>>>> is an example using pyczmq:
>>>>>
>>>>> >>> from pyczmq import zctx, zsocket, zmq
>>>>> >>> s = zctx.new()
>>>>> >>> zctx.underlying(s)  # None until a socket is made
>>>>> >>> p = zsocket.new(s, zmq.PUSH)
>>>>> >>> zmq.ctx_set(zctx.underlying(s), zmq.MAX_SOCKETS, 10)
>>>>> 0
>>>>> >>> zmq.ctx_get(zctx.underlying(s), zmq.MAX_SOCKETS)
>>>>> 10
>>>>> >>>
>>>>>
>>>>>
>>>>>
>>>>> On Mon, Jun 23, 2014 at 7:26 PM, Steve Murphy <murf at parsetree.com>
>>>>> wrote:
>>>>>
>>>>>> Philip--
>>>>>>
>>>>>> In the code you provided, you are mixing czmq
>>>>>> and zmq lib calls, and as stated, it won't work.
>>>>>>
>>>>>> The zctx_t that zctx_new provides is NOT a void*
>>>>>> that zmq_ctx_new() would give you, and the calls to zmq_ctx_set() and
>>>>>> .._get()
>>>>>> will not work properly if given a zctx_t. If you replace your call
>>>>>> to zctx_new() with zmq_ctx_new(), you will get better results.
>>>>>>
>>>>>> Now, Pieter mentioned that:
>>>>>>
>>>>>> "You can lower the max sockets per context, before creating your first
>>>>>> context. See zmq_ctx_set ()."
>>>>>>
>>>>>> But I think he meant to say "before creating your first *socket*."
>>>>>>
>>>>>>  (which, btw, is not in the ZMQ ref manual.)
>>>>>>
>>>>>> I double checked the zctx page in the CZMQ spec, and no function
>>>>>> is available to get/set the context options... at least the
>>>>>> MAX_SOCKETS, that is.
>>>>>> So, if you need to play with MAX_SOCKETS, you have to abandon CZMQ,
>>>>>> as there is no
>>>>>> way to slip from the zctx_t world to the void* world.
>>>>>>
>>>>>> And a quick look at max sockets on a new context shows that the
>>>>>> default for
>>>>>> a new context is 1023. in the 4.1 stuff, there is also a
>>>>>> ZMQ_SOCKET_LIMIT,
>>>>>> which is the absolute maximum you can set in a set() call, but this
>>>>>> isn't
>>>>>> in the 4.0 versions.
>>>>>>
>>>>>> murf
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Mon, Jun 23, 2014 at 10:25 AM, Philip Dizon <
>>>>>> philipdotdev at gmail.com> wrote:
>>>>>>
>>>>>>> Is it possible to set it ZMQ_MAX_SOCKETS for a czmq context object
>>>>>>> because I'm getting an assert failure when I do a zmq_ctx_get?
>>>>>>>
>>>>>>>  e.g.
>>>>>>>     client->ctx = zctx_new();
>>>>>>>     assert(client->ctx);
>>>>>>>
>>>>>>>     int max_sockets = 256;
>>>>>>>     zmq_ctx_set (client->ctx, ZMQ_MAX_SOCKETS, max_sockets);
>>>>>>>     assert (zmq_ctx_get (client->ctx, ZMQ_MAX_SOCKETS) ==
>>>>>>> max_sockets);
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On Thu, Jun 19, 2014 at 4:05 PM, Pieter Hintjens <ph at imatix.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> You can lower the max sockets per context, before creating your
>>>>>>>> first
>>>>>>>> context. See zmq_ctx_set ().
>>>>>>>>
>>>>>>>> On Fri, Jun 20, 2014 at 12:39 AM, Philip Dizon <
>>>>>>>> philipdotdev at gmail.com> wrote:
>>>>>>>> > Hi,
>>>>>>>> >
>>>>>>>> > I noticed that just creating a new zmq context and socket bumps
>>>>>>>> my memory
>>>>>>>> > usage by 18MB, and this is a big problem on my embedded system
>>>>>>>> which only
>>>>>>>> > contains about 100MB.  I determined this by using top command and
>>>>>>>> comparing
>>>>>>>> > the difference in mem usage.
>>>>>>>> > Is there any sort of option I can use to lower the memory usage?
>>>>>>>> >
>>>>>>>> > Thanks
>>>>>>>> >
>>>>>>>> > _______________________________________________
>>>>>>>> > zeromq-dev mailing list
>>>>>>>> > zeromq-dev at lists.zeromq.org
>>>>>>>> > http://lists.zeromq.org/mailman/listinfo/zeromq-dev
>>>>>>>> >
>>>>>>>> _______________________________________________
>>>>>>>> zeromq-dev mailing list
>>>>>>>> zeromq-dev at lists.zeromq.org
>>>>>>>> http://lists.zeromq.org/mailman/listinfo/zeromq-dev
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> _______________________________________________
>>>>>>> zeromq-dev mailing list
>>>>>>> zeromq-dev at lists.zeromq.org
>>>>>>> http://lists.zeromq.org/mailman/listinfo/zeromq-dev
>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>>
>>>>>> Steve Murphy
>>>>>> ParseTree Corporation
>>>>>> 57 Lane 17
>>>>>> Cody, WY 82414
>>>>>> ✉  murf at parsetree dot com
>>>>>> ☎ 307-899-5535
>>>>>>
>>>>>>
>>>>>>
>>>>>> _______________________________________________
>>>>>> zeromq-dev mailing list
>>>>>> zeromq-dev at lists.zeromq.org
>>>>>> http://lists.zeromq.org/mailman/listinfo/zeromq-dev
>>>>>>
>>>>>>
>>>>>
>>>>> _______________________________________________
>>>>> zeromq-dev mailing list
>>>>> zeromq-dev at lists.zeromq.org
>>>>> http://lists.zeromq.org/mailman/listinfo/zeromq-dev
>>>>>
>>>>>
>>>>
>>>>
>>>> --
>>>>
>>>> Steve Murphy
>>>> ParseTree Corporation
>>>> 57 Lane 17
>>>> Cody, WY 82414
>>>> ✉  murf at parsetree dot com
>>>> ☎ 307-899-5535
>>>>
>>>>
>>>>
>>>
>>>
>>> --
>>>
>>> Steve Murphy
>>> ParseTree Corporation
>>> 57 Lane 17
>>> Cody, WY 82414
>>> ✉  murf at parsetree dot com
>>> ☎ 307-899-5535
>>>
>>>
>>>
>>> _______________________________________________
>>> zeromq-dev mailing list
>>> zeromq-dev at lists.zeromq.org
>>> http://lists.zeromq.org/mailman/listinfo/zeromq-dev
>>>
>>>
>>
>> _______________________________________________
>> zeromq-dev mailing list
>> zeromq-dev at lists.zeromq.org
>> http://lists.zeromq.org/mailman/listinfo/zeromq-dev
>>
>>
> _______________________________________________
> zeromq-dev mailing list
> zeromq-dev at lists.zeromq.org
> http://lists.zeromq.org/mailman/listinfo/zeromq-dev
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.zeromq.org/pipermail/zeromq-dev/attachments/20140724/bfdc4620/attachment.htm>