[zeromq-dev] can I send messages to connect socket from multiple threads with mutex protection?

Thu Jan 3 17:30:16 CET 2019

That's correct. You can check in the header file, and you can see
what's DRAFT and what's not based on what is behind the _DRAFT ifdef.

On Thu, 2019-01-03 at 19:23 +0530, anand n wrote:
> Thanks Luca.
> 
> I am evaluating the new thread safe sockets 'ZMQ_SERVER & ZMQ_CLIENT'
> to
> use in my app. I see that these socket types are introduced in 4.2.0
> in
> draft state and I didn't find any update on these sockets in further
> release notes. So I assume that these are still in draft state and
> not
> declared as stable. Can someone please confirm?
> 
> Regards,
> Anand.
> 
> 
> 
> On Thu, Jan 3, 2019 at 4:42 PM Luca Boccassi <luca.boccassi at gmail.com
> >
> wrote:
> 
> > 1) The pair pattern has been stable for years and it's been used in
> > production and by multiple projects (it's what CZMQ's zactor uses)
> > so
> > it's fine
> > 
> > 2) The issue is that while your application might deploy locks and
> > mutexes, the background I/O thread does not, so you can get all
> > kinds
> > of contentions and subtle issues. The internal data structures are
> > designed and implemented to be safe for one reader and one writer.
> > 
> > On Thu, 2019-01-03 at 08:55 +0530, anand n wrote:
> > > Thanks a lot Bill, Juergen and Luca for your quick response and
> > > suggestions. I am looking at them.
> > > 
> > > I have few more questions/clarifications on the zmq-pair socket
> > > type
> > > (please see below) and want to say thanks in advance for your
> > > kind
> > > reply!
> > > 
> > > 1. I see that the word 'experimental' is dropped in the latest
> > > API
> > > documentation (http://api.zeromq.org/4-2:zmq-socket) for ZMQ_PAIR
> > > socket
> > > type. It just states that functionality such as 'auto-
> > > reconnection'
> > > is not
> > > implemented. So shall I safely assume that the socket type is
> > > official now
> > > and the basic functionality is expected to work fine without any
> > > issues?
> > > 
> > > 2. I understand that it is not recommended to use non-thread safe
> > > legacy
> > > zmq sockets in more than one thread. However just for my
> > > understanding, I
> > > am asking this question. Do you expect any problem from zmq
> > > implementation
> > > perspective, if the application shares a zmq-pair connect socket
> > > with
> > > multiple threads only to send message to a single peer thread
> > > (with
> > > mutex
> > > around zmq_send()) and doesn't use the socket in zmq_poll(). The
> > > socket
> > > will never be disconnected/destroyed by the application process
> > > once
> > > created and lives forever until the process exits.
> > > 
> > > I took a quick look at thread safe client-server socket type and
> > > it
> > > looks
> > > that it will address my requirement. I will explore more on that
> > > and
> > > come
> > > back if I have any question.
> > > 
> > > Thanks,
> > > Anand.
> > > 
> > > On Mon, Dec 31, 2018 at 9:42 PM Juergen Gnoss <jgnoss at hotmail.com
> > > >
> > > wrote:
> > > 
> > > > Hi Anand
> > > > 
> > > > Sounds you've already spent a fair amount of time in your
> > > > project,
> > > > but
> > > > anyway
> > > > here some of my experiences.
> > > > 
> > > > Time ago I was on the same point as you, and I gave the DRAFT
> > > > stuff
> > > > of CZMQ
> > > > a shot. Really exciting stuff that, and since then I kept using
> > > > that.
> > > > 
> > > > Putting all that API thread based stuff in zactors, keeping the
> > > > inproc
> > > > communication
> > > > on the internal pipe and put external communication on
> > > > additional
> > > > client/server sockets
> > > > in the actors gives a really clean design.
> > > > 
> > > > And yes, it's thread safe that way.
> > > > 
> > > > ju
> > > > 
> > > > ------------------------------
> > > > *From:* zeromq-dev <zeromq-dev-bounces at lists.zeromq.org> on
> > > > behalf
> > > > of
> > > > Bill Torpey <wallstprog at gmail.com>
> > > > *Sent:* Monday, December 31, 2018 10:55 AM
> > > > *To:* ZeroMQ development list
> > > > *Subject:* Re: [zeromq-dev] [C/Linux/ZMQ_PAIR/Unidirectional-
> > > > flow]
> > > > - can
> > > > I send messages to connect socket from multiple threads with
> > > > mutex
> > > > protection?
> > > > 
> > > > Hi Anand:
> > > > 
> > > > Please see comments below.  Good luck with your project!
> > > > 
> > > > Bill
> > > > 
> > > > > On Dec 30, 2018, at 10:45 PM, anand n <anand.pathivu at gmail.co
> > > > > m>
> > > > > wrote:
> > > > > 
> > > > > Team,
> > > > > 
> > > > > I am building a multi threaded application in linux with C as
> > > > > the
> > > > 
> > > > programming language and use ZMQ_PAIR sockets for inter-thread
> > > > communication (zmq version - 4.3.0). The main thread of the
> > > > application
> > > > communicates with external applications using router-dealer
> > > > model.
> > > > > 
> > > > > The application process spawns multiple child threads for
> > > > > handling
> > > > 
> > > > specific functionalities. Each thread creates two ZMQ_PAIR
> > > > sockets
> > > > (one for
> > > > bind & the other for connect) and bind & connect these sockets
> > > > to
> > > > same end
> > > > point (inproc://<unique-name-per-thread>). All these are done
> > > > during
> > > > process initialization and finally the threads invoke zmq_poll
> > > > on
> > > > the bind
> > > > socket. The connect socket is only used for sending messages to
> > > > the
> > > > thread
> > > > and the bind socket is used to receive them.
> > > > 
> > > > In my experience, PAIR sockets can be unreliable (
> > > > https://github.com/zeromq/libzmq/issues/2759).  I published
> > > > test
> > > > code
> > > > here (https://github.com/WallStProg/zmqtests/tree/master/thread
> > > > s)
> > > > that
> > > > demonstrates the problems.  What I ended up doing was using
> > > > CLIENT/SERVER
> > > > sockets instead of PAIR sockets, which have proved reliable in
> > > > my
> > > > testing.
> > > > 
> > > > Unlike PAIR sockets, SERVER sockets are not exclusive, so a
> > > > single
> > > > SERVER
> > > > socket can receive messages from any number of client sockets
> > > > (i.e., it is
> > > > one-to-many rather than one-to-one as with PAIR sockets).
> > > > 
> > > > Another benefit of CLIENT/SERVER sockets is that they are
> > > > thread-
> > > > safe,
> > > > which leads to my next comment…
> > > > 
> > > > > 
> > > > > The thread publishes APIs to other threads and messages are
> > > > > sent
> > > > > to the
> > > > 
> > > > connect socket of the thread from these APIs (in non-blocking
> > > > mode). Of
> > > > course socket access (i.e, zmq_send) is mutex protected from
> > > > application
> > > > perspective.
> > > > 
> > > > In general, protecting “legacy” ZMQ sockets with mutexes is not
> > > > recommended. While it may be theoretically possible to make
> > > > synchronization
> > > > work with mutexes, in practice that can be tricky, particularly
> > > > if
> > > > the
> > > > application uses zmq_poll to wait on socket events.
> > > > 
> > > > Better to restrict socket access to a single thread, or to use
> > > > one
> > > > of the
> > > > newer thread-safe socket types if possible.
> > > > 
> > > > > 
> > > > > I've tested the application and its working fine so far
> > > > > without
> > > > > any
> > > > 
> > > > issues. I understand that zmq socket is not thread safe and as
> > > > you
> > > > see, in
> > > > this model, messages are sent to the connect socket from
> > > > multiple
> > > > threads
> > > > (not concurrently anyway due to the mutex protection!).
> > > > 
> > > > I think you may be confusing endpoints and sockets — with some
> > > > socket
> > > > types (CLIENT/SERVER, PUB/SUB), many sockets can connect to a
> > > > single
> > > > endpoint, and in those cases any necessary synchronization is
> > > > handled
> > > > internally by ZMQ.
> > > > 
> > > > 
> > > > > Do you think that sharing the socket in this way can cause
> > > > > any
> > > > > problem
> > > > 
> > > > from zmq perspective?
> > > > > 
> > > > > Please let me know if something is not clear or if you need
> > > > > more
> > > > > info.
> > > > 
> > > > Thanks a lot for your help.
> > > > > 
> > > > > Regards,
> > > > > Anand.
> > > > > 
> > > > > _______________________________________________
> > > > > zeromq-dev mailing list
> > > > > zeromq-dev at lists.zeromq.org
> > > > > https://lists.zeromq.org/mailman/listinfo/zeromq-dev
> > > > 
> > > > _______________________________________________
> > > > zeromq-dev mailing list
> > > > zeromq-dev at lists.zeromq.org
> > > > https://lists.zeromq.org/mailman/listinfo/zeromq-dev
> > > > _______________________________________________
> > > > zeromq-dev mailing list
> > > > zeromq-dev at lists.zeromq.org
> > > > https://lists.zeromq.org/mailman/listinfo/zeromq-dev
> > > > 
> > > 
> > > _______________________________________________
> > > zeromq-dev mailing list
> > > zeromq-dev at lists.zeromq.org
> > > https://lists.zeromq.org/mailman/listinfo/zeromq-dev
> > 
> > --
> > Kind regards,
> > Luca Boccassi
> > _______________________________________________
> > zeromq-dev mailing list
> > zeromq-dev at lists.zeromq.org
> > https://lists.zeromq.org/mailman/listinfo/zeromq-dev
> > 
> 
> _______________________________________________
> zeromq-dev mailing list
> zeromq-dev at lists.zeromq.org
> https://lists.zeromq.org/mailman/listinfo/zeromq-dev

-- 
Kind regards,
Luca Boccassi
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 488 bytes
Desc: This is a digitally signed message part
URL: <https://lists.zeromq.org/pipermail/zeromq-dev/attachments/20190103/364adca0/attachment.sig>