[zeromq-dev] Problem: CLIENT/SERVER types cannot send multi-part messages.

Joe McIlvain joe.eli.mac at gmail.com
Sat Sep 12 00:45:53 CEST 2015


Solution: Add new libzmq API for sending and receiving multi-part messages
atomically/statelessly.

---

With this commit, the REQ, REP, DEALER, and ROUTER types are marked as
soon-to-be deprecated in favor of the CLIENT and SERVER types on libzmq
master:
https://github.com/hintjens/libzmq/commit/f3ee8c69dbb50322a931534a55940fef85ad71dd

This email is not about fighting the deprecation, but I do think that one
specific deficit in the current implementation of the CLIENT and SERVER
types: the lack of multi-part messages.

I think the lack of multi-part message support will limit the usefulness of
the CLIENT and SERVER types and as a consequence make it harder to
encourage adoption.  Also, I think multi-part messages are part of what
makes ZeroMQ incredibly convenient for rapid development of useful
applications.

Furthermore, the lack of multi-part messages seems entirely inspired by the
difficulty of creating thread-safe sockets within the C/C++ implementation
(libzmq).  This is because multi-part messages are sent using the
ZMQ_RCVMORE and ZMQ_SNDMORE flags, making the process a stateful one.
However, it strikes me that many high level bindings and ports are already
conceptually covering this statefulness with a stateless abstraction in
which multi-part messages are contained in a single object and sent with a
single API call (for example, zmsg in CZMQ contains multiple zframes).  In
such high level bindings and ports, removing multi-part messaging is quite
an artificial restriction, with no basis in conceptual technical necessity.

Because many such abstractions exist around the concept of multi-part
messages as a single object, both in bindings/ports and in applications,
removing support for multi-part messages in new socket types and
deprecating the old types that supports them creates a cascading effect of
rewrites that is likely to frustrate many maintainers across the spectrum.
So, to be sure, modern technologies will always undergo major changes if
they wish to remain modern, so this is not necessarily a bad thing, but
make no mistake that this design decision is quite a significant change in
the ZeroMQ ecosystem, even though it has been executed so far with minimal
impact to the API of libzmq.

I'd like to take a moment to consider the alternative - what if we could
make a more significant change to the API of libzmq in order to make a much
smaller impact on the broader ZeroMQ ecosystem, including the APIs of
bindings and ports as well as the multitude of applications?  What if we
can have both thread-safe sockets and keep multi-part messages - with the
requirement that bindings and applications desiring thread-safety use the
new atomic API for multi-part messages?  For existing applications using
the existing ZMQ_SNDMORE/ZMQ_RCVMORE API, nothing would change - they could
continue to use this (probably deprecated) API, but not if they wanted to
use the thread-safe socket types.  This is already the case - the new
ZMQ_CLIENT type will simply drop any messages with the ZMQ_SNDMORE flag (
https://github.com/zeromq/libzmq/blob/master/tests/test_client_drop_more.cpp#L55),
meaning that we're no worse off in this way.

I'm arguing that such an API, if done right would be:
- Backwards-compatible (the existing ZMQ_SNDMORE/ZMQ_RCVMORE API would
still exist, but probably be deprecated)
- Less of an impact on the broader ZeroMQ community than the current design
direction of deprecating multi-part messaging.
- Useful for making threadsafe implementations of the other patterns such
as the publish-subscribe pattern.

So far I fail to see any disadvantages that outweigh those advantages.

---

I'd like this email thread to be about discussing the design of such an API.

If we can come up with a coherent plan, I'd be happy to working on the
implementation, using the ZMQ_CLIENT and ZMQ_SERVER sockets as the guinea
pigs they already are.

One way to implement such an API that comes to mind, is with an approach
that mimics the `readv` and `writev` functions used in scatter/gather I/O (
http://linux.die.net/man/2/writev).  Such an API call in ZeroMQ could
accept a pointer and count for an array of pointers to `zmq_msg_t` objects,
instead of a pointer and count for an array of pointers to byte buffers.
Alternatively, a different struct definition other than `zmq_msg_t` could
be created that holds the pointer and count for an array of pointers to
byte buffers.  I think the first approach has a smaller API impact, but the
second approach is more elegant.  Both approaches would be preferable to
dropping support for multi-part messages.

Thoughts?
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.zeromq.org/pipermail/zeromq-dev/attachments/20150911/afcfa631/attachment.htm>


More information about the zeromq-dev mailing list