[zeromq-dev] Slow subscriber runs out of memory

[Rui Lopes]

Hello,

It seems to be already fixed at:

  https://github.com/zeromq/clrzmq/blob/master/src/clrzmq/Socket.cs#L234

        /// <summary>
        /// Set Socket Option
        /// </summary>
        /// <param name="option">Socket Option</param>
        /// <param name="value">Option value</param>
        /// <exception cref="ZMQ.Exception">ZMQ Exception</exception>
        public void SetSockOpt(SocketOpt option, ulong value) {
            int sizeOfValue = Marshal.SizeOf(typeof(ulong));
            using (var valPtr = new DisposableIntPtr(sizeOfValue)) {
                Marshal.WriteInt64(valPtr.Ptr,
unchecked(Convert.ToInt64(value)));
                if (C.zmq_setsockopt(Ptr, (int)option, valPtr.Ptr,
sizeOfValue) != 0)
                    throw new Exception();
            }
        }

Can you confirm?

Best regards,
Rui Lopes


On Tue, May 24, 2011 at 10:29, a frost <afrost408 at gmail.com> wrote:
> Hi,
>
> The problem is indeed in the clrzmq library and not in ZMQ itself. Setting
> HWM does not work correctly i.e. the wrong value is passed to the native ZMQ
> library.
> The value of HWM passed into setsockopt() must be a 64 bit value -
> regardless of whether the host OS is x86 or x64. The clrzmq .NET wrapper is
> only setting the lower 32 bits, instead of all 64 bits. This is due to a
> call to WriteSizeT() which in turn calls Marshal.WriteInt32()
>
> public void SetSockOpt(SocketOpt option, ulong value) {
> int sizeOfValue = Marshal.SizeOf(typeof(ulong));
> using (DisposableIntPtr valPtr = new DisposableIntPtr(sizeOfValue)) {
> WriteSizeT(valPtr.Ptr, value); // THIS IS WRONG - only the lower 32 bits get
> set, higher 32 bits contain garbage
> if (C.zmq_setsockopt(_ptr, (int)option, valPtr.Ptr, sizeOfValue) != 0)
> {
> throw new Exception();
> }
> }
> }
> The size of the pointer is 32 bits, but the value it points to is 64 bits
> i.e. __int64 on Windows. When I set the HWM to 10, the lower 32 bits
> contains 10, but the higher 32 bits contains garbage. So I end up with a
> huge value for HWM, which is always larger than the total memory on my PC.
> So setting the HWM has no practical limiting effect. As a result my
> subscriber process uses memory without bound.
> Not sure what the best solution is. Possibly the DisposableIntPtr class
> should have its own WriteIntXX() methods, since its this class that actually
> knows the size of the pointed to value?
>
> Maybe something like this (untested) code: http://pastebin.com/96CBMcYi
>
> Regards,
> Andy
>
> On Tue, May 24, 2011 at 12:28 AM, a frost <afrost408 at gmail.com> wrote:
>>
>> I think you are right. I have now ported to C and I don't get the memory
>> increase problem.
>>
>>
>> On Mon, May 23, 2011 at 11:58 PM, Joshua Foster <jhawk28 at gmail.com> wrote:
>>>
>>> I ported the code to Java and it does not increase in memory. My guess
>>> is that the clrzmq bindings are at fault.
>>>
>>> http://pastebin.com/nZACLqi1
>>>
>>> Joshua
>>>
>>> On 5/23/2011 11:35 AM, Martin Sustrik wrote:
>>> > On 05/23/2011 05:17 PM, a frost wrote:
>>> >
>>> >> I am evaluating ZMQ, doing some initial testing with a PUB-SUB
>>> >> configuration. The publisher and subscriber are in separate processes
>>> >> on
>>> >> the same machine. I am using TCP transport and have set HWM=10 on both
>>> >> sides.
>>> >>
>>> >> The subscriber is slow compared to the publisher and its memory usage
>>> >> increases rapidly as messages are received from the publisher.
>>> >>    Eventually, the subscriber process runs out of memory.
>>> >>
>>> >> I want messages to be discarded at the subscriber end if it cant
>>> >> process
>>> >> them fast enough, rather than memory usage increasing without bound as
>>> >> received messages are buffered. I thought setting HWM on the SUB side
>>> >> would give me this behaviour, but setting this has no effect on memory
>>> >> usage.
>>> > Yes. The messages should be discarded.
>>> >
>>> >> Any help would be appreciated. Source code for my test programs is at:
>>> >>
>>> >> Publisher: http://pastebin.com/2YtRAi8C
>>> >> Subscriber: http://pastebin.com/Sr16RP75
>>> > The programs look OK to me. I guess there is a bug either in 0MQ or
>>> > jzmq. I wonder if the same thing happens in C (ie. when jzmq is not
>>> > used).
>>> >
>>> > Martin
>>> > _______________________________________________
>>> > zeromq-dev mailing list
>>> > zeromq-dev at lists.zeromq.org
>>> > http://lists.zeromq.org/mailman/listinfo/zeromq-dev
>>>
>>> _______________________________________________
>>> zeromq-dev mailing list
>>> zeromq-dev at lists.zeromq.org
>>> http://lists.zeromq.org/mailman/listinfo/zeromq-dev
>>
>>
>
>
> _______________________________________________
> zeromq-dev mailing list
> zeromq-dev at lists.zeromq.org
> http://lists.zeromq.org/mailman/listinfo/zeromq-dev