[zeromq-dev] CZMQ zsys layer

Luca Boccassi luca.boccassi at gmail.com
Mon Jan 22 20:18:38 CET 2018


On Mon, 2018-01-22 at 15:41 +0000, Stephen Gray wrote:
> Hi everyone,
> 
> Lately I've been using CZMQ on Windows developing  .dlls as custom-
> user-extensions to a proprietary-app.   The proprietary-app is able
> to launch as many processes as it deems necessary;  each process will
> have a copy of the relevant dll and (I think) issue a copy to each
> thread as appropriate. The end result is multiple instances of
> classes using the CZMQ API, executing on different threads in the
> same process.
> 
> To my understanding there is only one instance of the CZMQ zsys_layer
> per process, no matter how many times zsys_init() gets called.
> 
> Here comes the tricky part......those multiple class instances need
> to come and go on their own terms.  If any one of them calls
> zsys_shutdown() then the zsys_layer instance serving all the other
> instances on threads in the process  will  be terminated and the
> system will crash.  If zsys_shutdown is not called at all then when
> the process terminates it will hang and be unsatisfactory.
> 
> With this setup I encountered termination/destruction issues which
> were somewhat tricky to isolate and solve; happily they are solved :)
> 
> I solved the issue by using
> boost::interprocess::managed_shared_memory with named-shared-memory-
> objects and the "open_or_create" functionality of
> boost::interprocess. With a boost::interprocess::named_mutex for good
> measure.  The shared memory segment is named using the value from
> GetCurrentProcess() from <windows.h>.  Therefore any thread of the
> same process can find the relevant shared memory.  In the shared
> memory is a std::atomic<int> holding the instance count.  At
> instantiation each thread increments the instance count, at
> termination each thread decrements the instance count.  If the
> instance count hits zero then it is safe to call
> zsys_shutdown().  Last one out; turns off the lights.
> 
> In the challenging environment of working all this out it struck me
> that the functionality and interface of zsys_shutdown could be
> improved to cater for other situations of similar multi-threaded
> complexity.
> 
> I would like to open the topic for discussion, but wonder if it best
> done here on the mailing list or better done on GitHub.  Not sure how
> really.
> 
> My feature suggestions are;
> 
> 
> 1.       Permit multiple instances of zsys_layer per-
> process.  Currently zsys_init() returns a pointer.  Use this pointer
> to permit targeted shutdown of a particular zsys_layer instance by
> changing the interface to; zsys_shutdown(ACCEPT_POINTER).

This would lead to breakages of actors and such, since inproc pipes
would not be able to communicate anymore between different instances.
Also there are (intentionally) a few static variables to store state
which would complicate things and require more extensive (and risky)
refactoring.

> 2.       Keep a record of user threads calling into the zsys_layer.
> On zys_shutdown if another user thread remains active then keep the
> zsys_layer operational. i.e. zsys_shutdown(SOFT_OPTION) would not
> have any effect in this circumstance.  Keep zsys_shutdown() as the
> HARD_WAY to brutally terminate the zsys_layer.

This should work fine and should be doable with relatively small
changes to src/zsys.c and api/zsys.api (if new APIs are added), feel
free to send a PR, but please make sure to:

1) Do not change the existing APIs - create new functions (eg:
zsys_shutdown_refcounted_only or something)
2) #ifdef it for Windows, since it's a workaround for a Windows-
specific issue, no need to make it more complicated for *nix

> Either of these options would negate the need for my shared memory
> work-around in this complex case.
> 
> If someone deeply familiar with the zsys_layer could comment on this
> it would be awesome.  My C skills aren't up to making a pull request
> though, I work in C++.
> 
> Thanks,
> 
> Stephen.

-- 
Kind regards,
Luca Boccassi
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 488 bytes
Desc: This is a digitally signed message part
URL: <https://lists.zeromq.org/pipermail/zeromq-dev/attachments/20180122/197429ef/attachment.sig>


More information about the zeromq-dev mailing list