[zeromq-dev] Thread safety for CLIENT and SERVER - help needed

[Doron Somech]

Hi All,

I'm trying to implement the thread safety for client and server sockets.

So far I did the safety for send and receive method and the code seems
right, however it doesn't behave exactly as I thought and I need help,
maybe I'm missing something, you can take a look at the change here (I left
the debug messages):

https://github.com/somdoron/libzmq/commit/403591007f61c2e9d1865be1bfe0349ec5c265d7

Little explanation of what I did:

1. When entering the recv or send method you also enter a mutex
2. If you need to wait for messages to receive / send queue is full you
exit the mutex and wait on the signaler (eventfd with poll).
3. When command is sent to the socket the signaler should be awaken, enter
the mutex again and try to send or receive.

The issue is with the second item, I wrote a test (test_thread_safe, you
can see in the commit) which two threads try to receive and response on
same socket, each wait 500ms after receive to let the other thread a chance
to receive the next message. The main thread always sends two messages and
then wait for two responses assuming one will come from each thread (500ms
is more then enough to let the other thread to be awaken), in practice it
doesn't work, I can see cases that it is taking a thread to be awaken few
seconds and only one thread is handling the messages. I set debug messages
around the internal of libzmq to try to understand, I can see that the
thread is blocked on waiting for command, however only one of the thread
(usually the same one) is awaken when command arrive and not both. Sometime
the other thread is not even on the waiting phase (on the sleep) and the
other thread is still not awaken.

Some of my thoughts:
* Two different threads are polling on the same file descriptor, from all
of my check this should be ok, also nanomsg is doing the same.
* Maybe the test project in libzmq is limited to one core? anyway i'm doing
a sleep, the operating system should try and process other threads.
* Maybe eventfd is not right for this?
* I have a bug?

I'm testing this on ubuntu desktop 14.10.

So I will appreciate any help.

Regarding if thread safe is right or not for zeromq, I think we have
another discussion for this, I will really appreciate help with the problem.

Doron
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.zeromq.org/pipermail/zeromq-dev/attachments/20150207/02e778ee/attachment.htm>