[zeromq-dev] Slow subscriber runs out of memory

[a frost]

Hi,

The problem is indeed in the clrzmq library and not in ZMQ itself. Setting
HWM does not work correctly i.e. the wrong value is passed to the native ZMQ
library.
The value of HWM passed into setsockopt() must be a 64 bit value -
regardless of whether the host OS is x86 or x64. The clrzmq .NET wrapper is
only setting the lower 32 bits, instead of all 64 bits. This is due to a
call to WriteSizeT() which in turn calls Marshal.WriteInt32()

public void SetSockOpt(SocketOpt option, ulong value) {
int sizeOfValue = Marshal.SizeOf(typeof(ulong));
using (DisposableIntPtr valPtr = new DisposableIntPtr(sizeOfValue)) {
* WriteSizeT(valPtr.Ptr, value); // THIS IS WRONG - only the lower 32 bits
get set, higher 32 bits contain garbage*
if (C.zmq_setsockopt(_ptr, (int)option, valPtr.Ptr, sizeOfValue) != 0)
{
throw new Exception();
}
}
}
The size of the pointer is 32 bits, but the value it points to is 64 bits
i.e. __int64 on Windows. When I set the HWM to 10, the lower 32 bits
contains 10, but the higher 32 bits contains garbage. So I end up with a
huge value for HWM, which is always larger than the total memory on my PC.
So setting the HWM has no practical limiting effect. As a result my
subscriber process uses memory without bound.
Not sure what the best solution is. Possibly the DisposableIntPtr class
should have its own WriteIntXX() methods, since its this class that actually
knows the size of the pointed to value?

Maybe something like this (untested) code: http://pastebin.com/96CBMcYi

Regards,
Andy

On Tue, May 24, 2011 at 12:28 AM, a frost <afrost408 at gmail.com> wrote:

> I think you are right. I have now ported to C and I don't get the memory
> increase problem.
>
>
> On Mon, May 23, 2011 at 11:58 PM, Joshua Foster <jhawk28 at gmail.com> wrote:
>
>> I ported the code to Java and it does not increase in memory. My guess
>> is that the clrzmq bindings are at fault.
>>
>> http://pastebin.com/nZACLqi1
>>
>> Joshua
>>
>> On 5/23/2011 11:35 AM, Martin Sustrik wrote:
>> > On 05/23/2011 05:17 PM, a frost wrote:
>> >
>> >> I am evaluating ZMQ, doing some initial testing with a PUB-SUB
>> >> configuration. The publisher and subscriber are in separate processes
>> on
>> >> the same machine. I am using TCP transport and have set HWM=10 on both
>> >> sides.
>> >>
>> >> The subscriber is slow compared to the publisher and its memory usage
>> >> increases rapidly as messages are received from the publisher.
>> >>    Eventually, the subscriber process runs out of memory.
>> >>
>> >> I want messages to be discarded at the subscriber end if it cant
>> process
>> >> them fast enough, rather than memory usage increasing without bound as
>> >> received messages are buffered. I thought setting HWM on the SUB side
>> >> would give me this behaviour, but setting this has no effect on memory
>> >> usage.
>> > Yes. The messages should be discarded.
>> >
>> >> Any help would be appreciated. Source code for my test programs is at:
>> >>
>> >> Publisher: http://pastebin.com/2YtRAi8C
>> >> Subscriber: http://pastebin.com/Sr16RP75
>> > The programs look OK to me. I guess there is a bug either in 0MQ or
>> > jzmq. I wonder if the same thing happens in C (ie. when jzmq is not
>> used).
>> >
>> > Martin
>> > _______________________________________________
>> > zeromq-dev mailing list
>> > zeromq-dev at lists.zeromq.org
>> > http://lists.zeromq.org/mailman/listinfo/zeromq-dev
>>
>> _______________________________________________
>> zeromq-dev mailing list
>> zeromq-dev at lists.zeromq.org
>> http://lists.zeromq.org/mailman/listinfo/zeromq-dev
>>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.zeromq.org/pipermail/zeromq-dev/attachments/20110524/2d097c6e/attachment.htm>