[zeromq-dev] can I send messages to connect socket from multiple threads with mutex protection?

anand n anand.pathivu at gmail.com
Thu Jan 3 14:53:06 CET 2019


Thanks Luca.

I am evaluating the new thread safe sockets 'ZMQ_SERVER & ZMQ_CLIENT' to
use in my app. I see that these socket types are introduced in 4.2.0 in
draft state and I didn't find any update on these sockets in further
release notes. So I assume that these are still in draft state and not
declared as stable. Can someone please confirm?

Regards,
Anand.



On Thu, Jan 3, 2019 at 4:42 PM Luca Boccassi <luca.boccassi at gmail.com>
wrote:

> 1) The pair pattern has been stable for years and it's been used in
> production and by multiple projects (it's what CZMQ's zactor uses) so
> it's fine
>
> 2) The issue is that while your application might deploy locks and
> mutexes, the background I/O thread does not, so you can get all kinds
> of contentions and subtle issues. The internal data structures are
> designed and implemented to be safe for one reader and one writer.
>
> On Thu, 2019-01-03 at 08:55 +0530, anand n wrote:
> > Thanks a lot Bill, Juergen and Luca for your quick response and
> > suggestions. I am looking at them.
> >
> > I have few more questions/clarifications on the zmq-pair socket type
> > (please see below) and want to say thanks in advance for your kind
> > reply!
> >
> > 1. I see that the word 'experimental' is dropped in the latest API
> > documentation (http://api.zeromq.org/4-2:zmq-socket) for ZMQ_PAIR
> > socket
> > type. It just states that functionality such as 'auto-reconnection'
> > is not
> > implemented. So shall I safely assume that the socket type is
> > official now
> > and the basic functionality is expected to work fine without any
> > issues?
> >
> > 2. I understand that it is not recommended to use non-thread safe
> > legacy
> > zmq sockets in more than one thread. However just for my
> > understanding, I
> > am asking this question. Do you expect any problem from zmq
> > implementation
> > perspective, if the application shares a zmq-pair connect socket with
> > multiple threads only to send message to a single peer thread (with
> > mutex
> > around zmq_send()) and doesn't use the socket in zmq_poll(). The
> > socket
> > will never be disconnected/destroyed by the application process once
> > created and lives forever until the process exits.
> >
> > I took a quick look at thread safe client-server socket type and it
> > looks
> > that it will address my requirement. I will explore more on that and
> > come
> > back if I have any question.
> >
> > Thanks,
> > Anand.
> >
> > On Mon, Dec 31, 2018 at 9:42 PM Juergen Gnoss <jgnoss at hotmail.com>
> > wrote:
> >
> > > Hi Anand
> > >
> > > Sounds you've already spent a fair amount of time in your project,
> > > but
> > > anyway
> > > here some of my experiences.
> > >
> > > Time ago I was on the same point as you, and I gave the DRAFT stuff
> > > of CZMQ
> > > a shot. Really exciting stuff that, and since then I kept using
> > > that.
> > >
> > > Putting all that API thread based stuff in zactors, keeping the
> > > inproc
> > > communication
> > > on the internal pipe and put external communication on additional
> > > client/server sockets
> > > in the actors gives a really clean design.
> > >
> > > And yes, it's thread safe that way.
> > >
> > > ju
> > >
> > > ------------------------------
> > > *From:* zeromq-dev <zeromq-dev-bounces at lists.zeromq.org> on behalf
> > > of
> > > Bill Torpey <wallstprog at gmail.com>
> > > *Sent:* Monday, December 31, 2018 10:55 AM
> > > *To:* ZeroMQ development list
> > > *Subject:* Re: [zeromq-dev] [C/Linux/ZMQ_PAIR/Unidirectional-flow]
> > > - can
> > > I send messages to connect socket from multiple threads with mutex
> > > protection?
> > >
> > > Hi Anand:
> > >
> > > Please see comments below.  Good luck with your project!
> > >
> > > Bill
> > >
> > > > On Dec 30, 2018, at 10:45 PM, anand n <anand.pathivu at gmail.com>
> > > > wrote:
> > > >
> > > > Team,
> > > >
> > > > I am building a multi threaded application in linux with C as the
> > >
> > > programming language and use ZMQ_PAIR sockets for inter-thread
> > > communication (zmq version - 4.3.0). The main thread of the
> > > application
> > > communicates with external applications using router-dealer model.
> > > >
> > > > The application process spawns multiple child threads for
> > > > handling
> > >
> > > specific functionalities. Each thread creates two ZMQ_PAIR sockets
> > > (one for
> > > bind & the other for connect) and bind & connect these sockets to
> > > same end
> > > point (inproc://<unique-name-per-thread>). All these are done
> > > during
> > > process initialization and finally the threads invoke zmq_poll on
> > > the bind
> > > socket. The connect socket is only used for sending messages to the
> > > thread
> > > and the bind socket is used to receive them.
> > >
> > > In my experience, PAIR sockets can be unreliable (
> > > https://github.com/zeromq/libzmq/issues/2759).  I published test
> > > code
> > > here (https://github.com/WallStProg/zmqtests/tree/master/threads)
> > > that
> > > demonstrates the problems.  What I ended up doing was using
> > > CLIENT/SERVER
> > > sockets instead of PAIR sockets, which have proved reliable in my
> > > testing.
> > >
> > > Unlike PAIR sockets, SERVER sockets are not exclusive, so a single
> > > SERVER
> > > socket can receive messages from any number of client sockets
> > > (i.e., it is
> > > one-to-many rather than one-to-one as with PAIR sockets).
> > >
> > > Another benefit of CLIENT/SERVER sockets is that they are thread-
> > > safe,
> > > which leads to my next comment…
> > >
> > > >
> > > > The thread publishes APIs to other threads and messages are sent
> > > > to the
> > >
> > > connect socket of the thread from these APIs (in non-blocking
> > > mode). Of
> > > course socket access (i.e, zmq_send) is mutex protected from
> > > application
> > > perspective.
> > >
> > > In general, protecting “legacy” ZMQ sockets with mutexes is not
> > > recommended. While it may be theoretically possible to make
> > > synchronization
> > > work with mutexes, in practice that can be tricky, particularly if
> > > the
> > > application uses zmq_poll to wait on socket events.
> > >
> > > Better to restrict socket access to a single thread, or to use one
> > > of the
> > > newer thread-safe socket types if possible.
> > >
> > > >
> > > > I've tested the application and its working fine so far without
> > > > any
> > >
> > > issues. I understand that zmq socket is not thread safe and as you
> > > see, in
> > > this model, messages are sent to the connect socket from multiple
> > > threads
> > > (not concurrently anyway due to the mutex protection!).
> > >
> > > I think you may be confusing endpoints and sockets — with some
> > > socket
> > > types (CLIENT/SERVER, PUB/SUB), many sockets can connect to a
> > > single
> > > endpoint, and in those cases any necessary synchronization is
> > > handled
> > > internally by ZMQ.
> > >
> > >
> > > > Do you think that sharing the socket in this way can cause any
> > > > problem
> > >
> > > from zmq perspective?
> > > >
> > > > Please let me know if something is not clear or if you need more
> > > > info.
> > >
> > > Thanks a lot for your help.
> > > >
> > > > Regards,
> > > > Anand.
> > > >
> > > > _______________________________________________
> > > > zeromq-dev mailing list
> > > > zeromq-dev at lists.zeromq.org
> > > > https://lists.zeromq.org/mailman/listinfo/zeromq-dev
> > >
> > > _______________________________________________
> > > zeromq-dev mailing list
> > > zeromq-dev at lists.zeromq.org
> > > https://lists.zeromq.org/mailman/listinfo/zeromq-dev
> > > _______________________________________________
> > > zeromq-dev mailing list
> > > zeromq-dev at lists.zeromq.org
> > > https://lists.zeromq.org/mailman/listinfo/zeromq-dev
> > >
> >
> > _______________________________________________
> > zeromq-dev mailing list
> > zeromq-dev at lists.zeromq.org
> > https://lists.zeromq.org/mailman/listinfo/zeromq-dev
>
> --
> Kind regards,
> Luca Boccassi
> _______________________________________________
> zeromq-dev mailing list
> zeromq-dev at lists.zeromq.org
> https://lists.zeromq.org/mailman/listinfo/zeromq-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.zeromq.org/pipermail/zeromq-dev/attachments/20190103/47d207f3/attachment.htm>


More information about the zeromq-dev mailing list