[zeromq-dev] Assertion failure in mailbox.cpp

[Dario Meloni]

On Thu, 02 Apr 2015 09:41:24 -0500, Thomas Rodgers wrote:

> can you get a stacktrace?
> 
> On Thu, Apr 2, 2015 at 9:27 AM, Dario Meloni <mellon85 at gmail.com> wrote:
> 
> 
>>
>> On Tue, 27 May 2014 10:55:57 +0100, Jon Gjengset wrote:
>>
>> I have this problem constantly in a software I am developing in my
>> company with ZMQ 4.0.4.
>> The issue arise in the same situation, one thread reading and one
>> writing in socket pairs inproc. When stress testing the application the
>> assertion will trigger sooner or later, but on reading from the
>> sockets, not shutting down.

To sum everything up

There are 2 threads and 2 connections.

Thread A:
A.pair.bind(inproc)          <-- ZMQ_PAIR

Thread B:
B.pair.connect(A.pair)       <-- ZMQ_PAIR
B.dealer.connect(tcp://.../) <-- ZMQ_DEALER


A.pair, B.pair do bidirectional communications between backend and 
frontend,
while B.dealers deals with the external world.  All communications are 
with
ZMQ_DONTWAIT. in case the queues are full the messages are dropped.

A uses libevent to listen on various stuff and the filedescriptor of the 
socket,
starts to read when the event has triggered.  A.pair and B.pair threads 
are
writing and reading from both at all times, but the each thread does only 
one of
the 2 at a time and only on their end of the socket.
All the code uses zmq_msg_send/recv.

I never had any assertion or error coming from B.dealer, and the problem 
arises
only when there is a lot of traffic going on.

 ** Sample 1 (mailbox:82)

#0  zmq::zmq_abort (errmsg_=errmsg_ at entry=0x7ffff795e3e3 "ok") at 
err.cpp:65
#1  0x00007ffff79342d5 in zmq::mailbox_t::recv (this=this at entry=0x689c40, 
cmd_=cmd_ at entry=0x7fffffffdf90, timeout_=timeout_ at entry=0) at 
mailbox.cpp:82
#2  0x00007ffff7948b29 in zmq::socket_base_t::process_commands 
(this=this at entry=0x689940, timeout_=timeout_ at entry=0, 
throttle_=throttle_ at entry=false) at socket_base.cpp:899
#3  0x00007ffff794915c in zmq::socket_base_t::recv (this=0x689940, 
msg_=msg_ at entry=0x7fffffffe090, flags_=<optimized out>) at 
socket_base.cpp:786
#4  0x00007ffff795ccd9 in s_recvmsg (s_=<optimized out>, 
msg_=0x7fffffffe090, flags_=<optimized out>) at zmq.cpp:460
#5  0x000000000042cb9b in (anonymous namespace)::getNewUrl 
(sock=0x689940) at file:488 

A.pair is receiving a message from B.pair

 ** Sample 2 (mailbox:82)

#0  zmq::zmq_abort (errmsg_=errmsg_ at entry=0x7ffff795e3e3 "ok") at 
err.cpp:65
#1  0x00007ffff79342d5 in zmq::mailbox_t::recv (this=this at entry=0x689c40, 
cmd_=cmd_ at entry=0x7ffff0719a50, timeout_=timeout_ at entry=0) at 
mailbox.cpp:82
#2  0x00007ffff7948b29 in zmq::socket_base_t::process_commands 
(this=this at entry=0x689940, timeout_=timeout_ at entry=0, 
throttle_=throttle_ at entry=true) at socket_base.cpp:899
#3  0x00007ffff7948e3b in zmq::socket_base_t::send 
(this=this at entry=0x689940, msg_=msg_ at entry=0x7ffff0719b20, 
flags_=flags_ at entry=1) at socket_base.cpp:692
#4  0x00007ffff795c8ac in s_sendmsg (s_=0x689940, msg_=0x7ffff0719b20, 
flags_=1) at zmq.cpp:350
#5  0x000000000042d7d1 in bar (this=this at entry=0x651198, id=..., 
reason=..., message=...) at file:196
#6  0x00000000004300fd in foo (this=0x651198) at file2:419

A.pair is sending to B.pair

Sometimes I also have data corruption instead of the assert. I am passing 
a
pointer but what's received on the other side is random junk and causes 
funny
segmentation faults.

I am also noticing more assertions popping up now that I am saving the 
stack
traces

 ** Sample 3

#0  zmq::zmq_abort (errmsg_=errmsg_ at entry=0x7ffff6985408 "Resource 
temporarily unavailable") at err.cpp:65
#1  0x00007ffff7947c1e in zmq::signaler_t::recv 
(this=this at entry=0x689ca0) at signaler.cpp:236
#2  0x00007ffff7934271 in zmq::mailbox_t::recv (this=this at entry=0x689c40, 
cmd_=cmd_ at entry=0x7fffffffdf30, timeout_=timeout_ at entry=0) at 
mailbox.cpp:68
#3  0x00007ffff7948b29 in zmq::socket_base_t::process_commands 
(this=this at entry=0x689940, timeout_=timeout_ at entry=0, 
throttle_=throttle_ at entry=true) at socket_base.cpp:899
#4  0x00007ffff7948e3b in zmq::socket_base_t::send 
(this=this at entry=0x689940, msg_=msg_ at entry=0x7fffffffe000, 
flags_=flags_ at entry=1) at socket_base.cpp:692
#5  0x00007ffff795c8ac in s_sendmsg (s_=0x689940, msg_=0x7fffffffe000, 
flags_=1) at zmq.cpp:350
#6  0x000000000042d7d1 in foo (this=this at entry=0x651198, id=..., 
reason=..., message=...) at file:196

A.pair is sending to B.pair

 ** Sample 4 

#0  zmq::zmq_abort (errmsg_=errmsg_ at entry=0x7ffff795ee85 "check ()") at 
err.cpp:65
#1  0x00007ffff79362a6 in zmq::msg_t::size 
(this=this at entry=0x7ffff0719e00) at msg.cpp:220
#2  0x00007ffff795c875 in zmq_msg_size (msg_=msg_ at entry=0x7ffff0719e00) 
at zmq.cpp:630
#3  0x00007ffff795cce5 in s_recvmsg (s_=<optimized out>, 
msg_=0x7ffff0719e00, flags_=<optimized out>) at zmq.cpp:463
#4  0x000000000042d0ec in bar (sock=0x68b380) at file:489
#5  0x000000000042faef in foo (this=0x651198) at file:315

B.pair reads from A.pair in an empty, initialized message