[zeromq-dev] content based routing for a thread pool

[Brett Viren]

Lei Jiang <lei.jiang.29 at gmail.com> writes:

> While putting together my code, I find it a bit hard to keep the implementation simple and efficient. For
> example, the routing IDs are separated by empty frames as delimiters. According to some charts in the guide,
> multiple IDs could be separated by multiple delimiters. This makes it impossible to identify message body
> efficiently until all frames have been received

This is a feature.  Multiple layers of routing can be accommodated by a
sequence of these routing parts.    IIRC, the guide describes this as
nested "envelopes".  The first ROUTER need only care about the first
parts and should (must) leave any subsequent parts un-recv'ed.

>  (BTW there does not seem to be a formal documentation on the
> message formats?).

The "ZMQ_ROUTER" section of the zmq_socket(3) man page documents the
message schema that pertains to routing.  It is also defined by the
REPREQ RFC:

  https://rfc.zeromq.org/spec/28/

> Also it seems the send() calls will close the message being sent,
> which I find a bit odd, too.

zmq_msg_send(3) man page states this behavior and suggests to copy the
message object if you wish to send it multiple times.  (In general,
ZeroMQ is very good at stating these details in man pages and RFCs).

I don't know this internal detail myself but I have always assumed this
is done so that the socket internals may avoid a copy in the more common
case of sending a message to a single socket.

> Another thing I find is that it's quite hard to make good use of features like zero-copy and multi-part
> messages. When having hops, every hop will at least require an additional copy. For the multipart,
> originally I thought I could use it for responses with unknown lengths. But then I realized because the
> proxy code can't serve any other peer until current request or response is finished, it will for sure block
> everyone else. The only remedy I can think of is for every packet to carry ID or routing data instead of
> using the "more" flag. So in reality the best use of multipart is probably to carry different parts/fields
> of requests/responses.

Except for the requirements of ROUTER, the application is free to do
what it wants in defining messages schema.

In making such definitions one contends with how explicit or implicit to
get.  Eg, should the message schema have some amount of self-description
or are endpoints more trusted to "do the right thing".

To give you a flavor, here is an example of an "opinionated" message
format I've used in the past:

  https://brettviren.github.io/zio/messages.html

Warning: this format is NOT at all as well thought out as it should be.
It also has some application-specific parts at levels that makes it not
generally useful.

> Lastly, regarding the proxy being single threaded, I think that's a
> must? As the stable versions don't support accessing sockets from
> different threads, the bridging code of any 2 sockets must run in a
> single thread.

Yes.

But a "proxy" is typically not doing much work other than:

  recv() -> [marshal] -> send()

If the "[marshal]" code becomes so computationally expensive as to
require multiple threads in order to keep up with the message rate from
the proxy's clients then perhaps the architecture needs rethinking.

For example, the "[marshal]" code can become a gateway to multi-threaded
(or even multi-process) workers.  Eg, "[marshal]" could be made to be
also a "client" in a MDP/PPP pattern.  Such "[marshal]" code would best
to use a poller watching both its own client socket and the socket used
to talk to the MDP/PPP "broker".  Then, the code inside the [marshal]'s
main loop that is servicing that poller will become a rather fast path
that can run in a single thread.


-Brett.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 849 bytes
Desc: not available
URL: <https://lists.zeromq.org/pipermail/zeromq-dev/attachments/20231030/1da564bb/attachment.sig>