[zeromq-dev] PUSH does not seem to release PULL connection message memory

[Greg Rogers]

On 2014-11-11 23:56:27 +0000, Stefan de Konink said:

> On Monday, November 10, 2014 6:01:27 PM CEST, Marc Norton wrote:
>> I expected that the RES memory would stabilize .... Perhaps I
>> don't understand something basic here - hence the question.
>> Note that if I comment out the usleep() call in pull.c  the
>> memory does not grow.
> 
> I am under the impression that the following bug is quite related;
> 
> <https://github.com/zeromq/libzmq/issues/1256>
> 
> I notice that zeromq does release memory in the context of CZMQ, but a
> naive C implementation clearly show at what point the memory has been
> allocated. Pieter suggested that: "It's possible the pub socket doesn't see
> the TCP connection closing, yes."
> 
> Stefan

I've been working on characterizing this bug.
It is not affected by the linger setting, -1, 0, 1000.
It is affected by the high water mark on both PUSH and PULL side but 
only as a HWM of 1 keeps it from being a problem, nothing in queues but 
HWM > 1 determines the growth with every disconnect.
The leak happens wether the pull socket is properly closed or just cut off.
So far have verifies the leak exists for PUSH/PULL and PUB/SUB.

Tracing through the code I have found that in this situation the pipe_t 
for the connection is never shutdown.

session_base_t::process_term() is called and it calls pipe_t::terminate()
The pipe_t term command is sent and received, the delimiter is sent but 
is never received so it has one sub pipe waiting on the delimiter and 
the other waiting on the term ack. It then stays in this state until 
the context is destroyed.

Any new connections form a new pipe in the lb_t and the half closed one 
is just marked as inactive.

I have been testing this with HWM on both ends set at 40,000 to ensure 
the memory leak is visible.

Would like to get some guidance on a fix, work around or where to dig further?

I have been adding a bunch of tracing statements to libzmq:
virtual void zmq::session_base_t::process_term(int):397
linger_(1000) > 0
virtual void zmq::session_base_t::process_term(int):423 linger_ = 1000
void zmq::pipe_t::terminate(bool):379 delay_ 1
void zmq::pipe_t::terminate(bool):380 delay  1
void zmq::pipe_t::terminate(bool):402 sending term
void zmq::pipe_t::terminate(bool):441 rollback
void zmq::pipe_t::rollback():221
void zmq::pipe_t::rollback():225
void zmq::pipe_t::terminate(bool):449 writing delimiter
void zmq::pipe_t::terminate(bool):451 1
void zmq::pipe_t::terminate(bool):455 leaving
virtual void zmq::session_base_t::process_term(int):425 back from terminate
bool zmq::pipe_t::check_read():119
bool zmq::pipe_t::check_read():130
not active and not waiting for delimiter
virtual void zmq::session_base_t::process_term(int):431 back from check read
virtual void zmq::pipe_t::process_pipe_term():289 delay = 1
virtual void zmq::pipe_t::process_pipe_term():296 was active
virtual void zmq::pipe_t::process_pipe_term():303 waiting for delimiter
virtual void zmq::session_base_t::timer_event(int):450
void zmq::pipe_t::terminate(bool):379 delay_ 0
void zmq::pipe_t::terminate(bool):380 delay  1
void zmq::pipe_t::terminate(bool):386 req sent