[zeromq-dev] [C/Linux/ZMQ_PAIR/Unidirectional-flow] - can I send messages to connect socket from multiple threads with mutex protection?

Bill Torpey wallstprog at gmail.com
Mon Dec 31 16:55:03 CET 2018


Hi Anand:

Please see comments below.  Good luck with your project!

Bill

> On Dec 30, 2018, at 10:45 PM, anand n <anand.pathivu at gmail.com> wrote:
> 
> Team,
> 
> I am building a multi threaded application in linux with C as the programming language and use ZMQ_PAIR sockets for inter-thread communication (zmq version - 4.3.0). The main thread of the application communicates with external applications using router-dealer model.
> 
> The application process spawns multiple child threads for handling specific functionalities. Each thread creates two ZMQ_PAIR sockets (one for bind & the other for connect) and bind & connect these sockets to same end point (inproc://<unique-name-per-thread>). All these are done during process initialization and finally the threads invoke zmq_poll on the bind socket. The connect socket is only used for sending messages to the thread and the bind socket is used to receive them. 

In my experience, PAIR sockets can be unreliable (https://github.com/zeromq/libzmq/issues/2759).  I published test code here (https://github.com/WallStProg/zmqtests/tree/master/threads) that demonstrates the problems.  What I ended up doing was using CLIENT/SERVER sockets instead of PAIR sockets, which have proved reliable in my testing.  

Unlike PAIR sockets, SERVER sockets are not exclusive, so a single SERVER socket can receive messages from any number of client sockets (i.e., it is one-to-many rather than one-to-one as with PAIR sockets).

Another benefit of CLIENT/SERVER sockets is that they are thread-safe, which leads to my next comment…

> 
> The thread publishes APIs to other threads and messages are sent to the connect socket of the thread from these APIs (in non-blocking mode). Of course socket access (i.e, zmq_send) is mutex protected from application perspective.

In general, protecting “legacy” ZMQ sockets with mutexes is not recommended. While it may be theoretically possible to make synchronization work with mutexes, in practice that can be tricky, particularly if the application uses zmq_poll to wait on socket events.

Better to restrict socket access to a single thread, or to use one of the newer thread-safe socket types if possible.

> 
> I've tested the application and its working fine so far without any issues. I understand that zmq socket is not thread safe and as you see, in this model, messages are sent to the connect socket from multiple threads (not concurrently anyway due to the mutex protection!).

I think you may be confusing endpoints and sockets — with some socket types (CLIENT/SERVER, PUB/SUB), many sockets can connect to a single endpoint, and in those cases any necessary synchronization is handled internally by ZMQ.  


> Do you think that sharing the socket in this way can cause any problem from zmq perspective?
> 
> Please let me know if something is not clear or if you need more info. Thanks a lot for your help.
> 
> Regards,
> Anand.
> 
> _______________________________________________
> zeromq-dev mailing list
> zeromq-dev at lists.zeromq.org
> https://lists.zeromq.org/mailman/listinfo/zeromq-dev




More information about the zeromq-dev mailing list