[zeromq-dev] can I send messages to connect socket from multiple threads with mutex protection?

[Luca Boccassi]

1) The pair pattern has been stable for years and it's been used in
production and by multiple projects (it's what CZMQ's zactor uses) so
it's fine

2) The issue is that while your application might deploy locks and
mutexes, the background I/O thread does not, so you can get all kinds
of contentions and subtle issues. The internal data structures are
designed and implemented to be safe for one reader and one writer.

On Thu, 2019-01-03 at 08:55 +0530, anand n wrote:
> Thanks a lot Bill, Juergen and Luca for your quick response and
> suggestions. I am looking at them.
> 
> I have few more questions/clarifications on the zmq-pair socket type
> (please see below) and want to say thanks in advance for your kind
> reply!
> 
> 1. I see that the word 'experimental' is dropped in the latest API
> documentation (http://api.zeromq.org/4-2:zmq-socket) for ZMQ_PAIR
> socket
> type. It just states that functionality such as 'auto-reconnection'
> is not
> implemented. So shall I safely assume that the socket type is
> official now
> and the basic functionality is expected to work fine without any
> issues?
> 
> 2. I understand that it is not recommended to use non-thread safe
> legacy
> zmq sockets in more than one thread. However just for my
> understanding, I
> am asking this question. Do you expect any problem from zmq
> implementation
> perspective, if the application shares a zmq-pair connect socket with
> multiple threads only to send message to a single peer thread (with
> mutex
> around zmq_send()) and doesn't use the socket in zmq_poll(). The
> socket
> will never be disconnected/destroyed by the application process once
> created and lives forever until the process exits.
> 
> I took a quick look at thread safe client-server socket type and it
> looks
> that it will address my requirement. I will explore more on that and
> come
> back if I have any question.
> 
> Thanks,
> Anand.
> 
> On Mon, Dec 31, 2018 at 9:42 PM Juergen Gnoss <jgnoss at hotmail.com>
> wrote:
> 
> > Hi Anand
> > 
> > Sounds you've already spent a fair amount of time in your project,
> > but
> > anyway
> > here some of my experiences.
> > 
> > Time ago I was on the same point as you, and I gave the DRAFT stuff
> > of CZMQ
> > a shot. Really exciting stuff that, and since then I kept using
> > that.
> > 
> > Putting all that API thread based stuff in zactors, keeping the
> > inproc
> > communication
> > on the internal pipe and put external communication on additional
> > client/server sockets
> > in the actors gives a really clean design.
> > 
> > And yes, it's thread safe that way.
> > 
> > ju
> > 
> > ------------------------------
> > *From:* zeromq-dev <zeromq-dev-bounces at lists.zeromq.org> on behalf
> > of
> > Bill Torpey <wallstprog at gmail.com>
> > *Sent:* Monday, December 31, 2018 10:55 AM
> > *To:* ZeroMQ development list
> > *Subject:* Re: [zeromq-dev] [C/Linux/ZMQ_PAIR/Unidirectional-flow]
> > - can
> > I send messages to connect socket from multiple threads with mutex
> > protection?
> > 
> > Hi Anand:
> > 
> > Please see comments below.  Good luck with your project!
> > 
> > Bill
> > 
> > > On Dec 30, 2018, at 10:45 PM, anand n <anand.pathivu at gmail.com>
> > > wrote:
> > > 
> > > Team,
> > > 
> > > I am building a multi threaded application in linux with C as the
> > 
> > programming language and use ZMQ_PAIR sockets for inter-thread
> > communication (zmq version - 4.3.0). The main thread of the
> > application
> > communicates with external applications using router-dealer model.
> > > 
> > > The application process spawns multiple child threads for
> > > handling
> > 
> > specific functionalities. Each thread creates two ZMQ_PAIR sockets
> > (one for
> > bind & the other for connect) and bind & connect these sockets to
> > same end
> > point (inproc://<unique-name-per-thread>). All these are done
> > during
> > process initialization and finally the threads invoke zmq_poll on
> > the bind
> > socket. The connect socket is only used for sending messages to the
> > thread
> > and the bind socket is used to receive them.
> > 
> > In my experience, PAIR sockets can be unreliable (
> > https://github.com/zeromq/libzmq/issues/2759).  I published test
> > code
> > here (https://github.com/WallStProg/zmqtests/tree/master/threads)
> > that
> > demonstrates the problems.  What I ended up doing was using
> > CLIENT/SERVER
> > sockets instead of PAIR sockets, which have proved reliable in my
> > testing.
> > 
> > Unlike PAIR sockets, SERVER sockets are not exclusive, so a single
> > SERVER
> > socket can receive messages from any number of client sockets
> > (i.e., it is
> > one-to-many rather than one-to-one as with PAIR sockets).
> > 
> > Another benefit of CLIENT/SERVER sockets is that they are thread-
> > safe,
> > which leads to my next comment…
> > 
> > > 
> > > The thread publishes APIs to other threads and messages are sent
> > > to the
> > 
> > connect socket of the thread from these APIs (in non-blocking
> > mode). Of
> > course socket access (i.e, zmq_send) is mutex protected from
> > application
> > perspective.
> > 
> > In general, protecting “legacy” ZMQ sockets with mutexes is not
> > recommended. While it may be theoretically possible to make
> > synchronization
> > work with mutexes, in practice that can be tricky, particularly if
> > the
> > application uses zmq_poll to wait on socket events.
> > 
> > Better to restrict socket access to a single thread, or to use one
> > of the
> > newer thread-safe socket types if possible.
> > 
> > > 
> > > I've tested the application and its working fine so far without
> > > any
> > 
> > issues. I understand that zmq socket is not thread safe and as you
> > see, in
> > this model, messages are sent to the connect socket from multiple
> > threads
> > (not concurrently anyway due to the mutex protection!).
> > 
> > I think you may be confusing endpoints and sockets — with some
> > socket
> > types (CLIENT/SERVER, PUB/SUB), many sockets can connect to a
> > single
> > endpoint, and in those cases any necessary synchronization is
> > handled
> > internally by ZMQ.
> > 
> > 
> > > Do you think that sharing the socket in this way can cause any
> > > problem
> > 
> > from zmq perspective?
> > > 
> > > Please let me know if something is not clear or if you need more
> > > info.
> > 
> > Thanks a lot for your help.
> > > 
> > > Regards,
> > > Anand.
> > > 
> > > _______________________________________________
> > > zeromq-dev mailing list
> > > zeromq-dev at lists.zeromq.org
> > > https://lists.zeromq.org/mailman/listinfo/zeromq-dev
> > 
> > _______________________________________________
> > zeromq-dev mailing list
> > zeromq-dev at lists.zeromq.org
> > https://lists.zeromq.org/mailman/listinfo/zeromq-dev
> > _______________________________________________
> > zeromq-dev mailing list
> > zeromq-dev at lists.zeromq.org
> > https://lists.zeromq.org/mailman/listinfo/zeromq-dev
> > 
> 
> _______________________________________________
> zeromq-dev mailing list
> zeromq-dev at lists.zeromq.org
> https://lists.zeromq.org/mailman/listinfo/zeromq-dev

-- 
Kind regards,
Luca Boccassi