[zeromq-dev] 0MQ-based proxy worker crashes with "Assertion failed: pipe (bundled/zeromq/src/session_base.cpp:441)"

[Tomas Krajca]

Hi,

I've got a 0MQ-based proxy, clients talk 0MQ to the proxy, the proxy then
talks HTTP to do either a GET on a specific url endpoint or a POST on a
specific endpoint (it always goes to one of these two url endpoints). I've
got a master process that has a zmq.ROUTER towards its clients (zmq.REQ)
and a zmq.DEALER towards its workers (zmq.REP). The master is a single
process, no threading, normal 0MQ, it spawns a worker processes via
multiprocessing, this worker process uses gevent and zmq.green to spawn the
actual (green) workers (those use grequests to talk HTTP). The master uses
0MQ auth to authenticate its clients. It should all be pretty standard but
note that this is my first gevent/zmq.green based project.

So this proxy runs pretty well untill the worker process (I run only 1
worker process) crashes with 'Assertion failed: pipe
(bundled/zeromq/src/session_base.cpp:441)"' on its stderr. There is nothing
else in the logs or on stdout that would give me any more idea of what is
going on. I can see the master running and netcat to its zmq.ROUTER so its
definitely the worker that dies. Sorry, I have no idea how to reproduce
this, once it crashed after 5 hours of working nicely, second time it
crashed after about a day.

Here is a snippet of the worker code (the relevant bits):
http://pastebin.com/usi0FXDL

The STSDBResponder uses grequests to do the HTTP, there should be nothing
special about that.

This happens on CentOS 6.5, the proxy is running in virtualenv (pip install
pyzmq):

Python 2.7.6 (default, Jul 10 2014, 04:59:13)
[GCC 4.4.7 20120313 (Red Hat 4.4.7-4)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import zmq
>>> zmq.zmq_version()
'4.0.4'
>>> zmq.__version__
'14.3.1'
>>>

I have no idea whether this is a libzmq bug or pyzmq bug or a bug in my
code or a system misconfiguration (do I need to increase ulimit or
something?), I run 64 gevent threads. I tried to see session_base.cpp but
it didn't help me understand why this could happen either.

If anybody could please point me to a direction as to why the worker
crashes, it would be much appreciated.

Thanks,
Tomas
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.zeromq.org/pipermail/zeromq-dev/attachments/20140909/69b11add/attachment.htm>