[zeromq-dev] can I send messages to connect socket from multiple threads with mutex protection?

Juergen Gnoss jgnoss at hotmail.com
Thu Jan 3 10:54:53 CET 2019


The question is not
"Do you expect any problem ... the application shares a zmq-pair connect
socket with multiple threads"

The question is, Is it right to do that.
There the answer is clearly no.

You should do an application design by looking at the applications future.

I personally don't want to keep always the weak points of my (bad designed)
application in mind while I'm extending it.

ju

________________________________
From: zeromq-dev <zeromq-dev-bounces at lists.zeromq.org> on behalf of anand n <anand.pathivu at gmail.com>
Sent: Wednesday, January 2, 2019 10:25 PM
To: ZeroMQ development list
Subject: Re: [zeromq-dev] can I send messages to connect socket from multiple threads with mutex protection?

Thanks a lot Bill, Juergen and Luca for your quick response and suggestions. I am looking at them.

I have few more questions/clarifications on the zmq-pair socket type (please see below) and want to say thanks in advance for your kind reply!

1. I see that the word 'experimental' is dropped in the latest API documentation (http://api.zeromq.org/4-2:zmq-socket) for ZMQ_PAIR socket type. It just states that functionality such as 'auto-reconnection' is not implemented. So shall I safely assume that the socket type is official now and the basic functionality is expected to work fine without any issues?

2. I understand that it is not recommended to use non-thread safe legacy zmq sockets in more than one thread. However just for my understanding, I am asking this question. Do you expect any problem from zmq implementation perspective, if the application shares a zmq-pair connect socket with multiple threads only to send message to a single peer thread (with mutex around zmq_send()) and doesn't use the socket in zmq_poll(). The socket will never be disconnected/destroyed by the application process once created and lives forever until the process exits.

I took a quick look at thread safe client-server socket type and it looks that it will address my requirement. I will explore more on that and come back if I have any question.

Thanks,
Anand.

On Mon, Dec 31, 2018 at 9:42 PM Juergen Gnoss <jgnoss at hotmail.com<mailto:jgnoss at hotmail.com>> wrote:
Hi Anand

Sounds you've already spent a fair amount of time in your project, but anyway
here some of my experiences.

Time ago I was on the same point as you, and I gave the DRAFT stuff of CZMQ
a shot. Really exciting stuff that, and since then I kept using that.

Putting all that API thread based stuff in zactors, keeping the inproc communication
on the internal pipe and put external communication on additional client/server sockets
in the actors gives a really clean design.

And yes, it's thread safe that way.

ju

________________________________
From: zeromq-dev <zeromq-dev-bounces at lists.zeromq.org<mailto:zeromq-dev-bounces at lists.zeromq.org>> on behalf of Bill Torpey <wallstprog at gmail.com<mailto:wallstprog at gmail.com>>
Sent: Monday, December 31, 2018 10:55 AM
To: ZeroMQ development list
Subject: Re: [zeromq-dev] [C/Linux/ZMQ_PAIR/Unidirectional-flow] - can I send messages to connect socket from multiple threads with mutex protection?

Hi Anand:

Please see comments below.  Good luck with your project!

Bill

> On Dec 30, 2018, at 10:45 PM, anand n <anand.pathivu at gmail.com<mailto:anand.pathivu at gmail.com>> wrote:
>
> Team,
>
> I am building a multi threaded application in linux with C as the programming language and use ZMQ_PAIR sockets for inter-thread communication (zmq version - 4.3.0). The main thread of the application communicates with external applications using router-dealer model.
>
> The application process spawns multiple child threads for handling specific functionalities. Each thread creates two ZMQ_PAIR sockets (one for bind & the other for connect) and bind & connect these sockets to same end point (inproc://<unique-name-per-thread>). All these are done during process initialization and finally the threads invoke zmq_poll on the bind socket. The connect socket is only used for sending messages to the thread and the bind socket is used to receive them.

In my experience, PAIR sockets can be unreliable (https://github.com/zeromq/libzmq/issues/2759).  I published test code here (https://github.com/WallStProg/zmqtests/tree/master/threads) that demonstrates the problems.  What I ended up doing was using CLIENT/SERVER sockets instead of PAIR sockets, which have proved reliable in my testing.

Unlike PAIR sockets, SERVER sockets are not exclusive, so a single SERVER socket can receive messages from any number of client sockets (i.e., it is one-to-many rather than one-to-one as with PAIR sockets).

Another benefit of CLIENT/SERVER sockets is that they are thread-safe, which leads to my next comment…

>
> The thread publishes APIs to other threads and messages are sent to the connect socket of the thread from these APIs (in non-blocking mode). Of course socket access (i.e, zmq_send) is mutex protected from application perspective.

In general, protecting “legacy” ZMQ sockets with mutexes is not recommended. While it may be theoretically possible to make synchronization work with mutexes, in practice that can be tricky, particularly if the application uses zmq_poll to wait on socket events.

Better to restrict socket access to a single thread, or to use one of the newer thread-safe socket types if possible.

>
> I've tested the application and its working fine so far without any issues. I understand that zmq socket is not thread safe and as you see, in this model, messages are sent to the connect socket from multiple threads (not concurrently anyway due to the mutex protection!).

I think you may be confusing endpoints and sockets — with some socket types (CLIENT/SERVER, PUB/SUB), many sockets can connect to a single endpoint, and in those cases any necessary synchronization is handled internally by ZMQ.


> Do you think that sharing the socket in this way can cause any problem from zmq perspective?
>
> Please let me know if something is not clear or if you need more info. Thanks a lot for your help.
>
> Regards,
> Anand.
>
> _______________________________________________
> zeromq-dev mailing list
> zeromq-dev at lists.zeromq.org<mailto:zeromq-dev at lists.zeromq.org>
> https://lists.zeromq.org/mailman/listinfo/zeromq-dev

_______________________________________________
zeromq-dev mailing list
zeromq-dev at lists.zeromq.org<mailto:zeromq-dev at lists.zeromq.org>
https://lists.zeromq.org/mailman/listinfo/zeromq-dev
_______________________________________________
zeromq-dev mailing list
zeromq-dev at lists.zeromq.org<mailto:zeromq-dev at lists.zeromq.org>
https://lists.zeromq.org/mailman/listinfo/zeromq-dev
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.zeromq.org/pipermail/zeromq-dev/attachments/20190103/d5e799a8/attachment.htm>


More information about the zeromq-dev mailing list