[zeromq-dev] content based routing for a thread pool

[Lei Jiang]

Hi Brett,

I managed to bridge 2 router sockets and get what I wanted. Basically the
code will preserve the mapping from the client's ID to the requester's ID
and, swap the client ID with the worker ID for the requests; and swap the
client ID back for the responses.

While putting together my code, I find it a bit hard to keep the
implementation simple and efficient. For example, the routing IDs are
separated by empty frames as delimiters. According to some charts in the
guide, multiple IDs could be separated by multiple delimiters. This makes
it impossible to identify message body efficiently until all frames have
been received (BTW there does not seem to be a formal documentation on the
message formats?). Also it seems the send() calls will close the message
being sent, which I find a bit odd, too.

Another thing I find is that it's quite hard to make good use of features
like zero-copy and multi-part messages. When having hops, every hop will at
least require an additional copy. For the multipart, originally I thought I
could use it for responses with unknown lengths. But then I realized
because the proxy code can't serve any other peer until current request or
response is finished, it will for sure block everyone else. The only remedy
I can think of is for every packet to carry ID or routing data instead of
using the "more" flag. So in reality the best use of multipart is probably
to carry different parts/fields of requests/responses.

Lastly, regarding the proxy being single threaded, I think that's a must?
As the stable versions don't support accessing sockets from different
threads, the bridging code of any 2 sockets must run in a single thread.


Cheers,
Lei



On Thu, Oct 26, 2023 at 1:17 AM Brett Viren <bv at bnl.gov> wrote:

> Lei Jiang <lei.jiang.29 at gmail.com> writes:
>
> > Looks like I need to understand the MDP in better detail. I will give
> > current design another try before exploring the MDP/PPP. My current
> > plan is to "game" the router socket to route to the peer I want. For
> > that I will maintain my own mappings. A list of all peer routing IDs
> > of the REP sockets, and a map from the REP routing IDs to REQ routing
> > IDs for all active requests.
>
> I think you can make something like that work.
>
> If you find this "condensed" design becomes unwieldy you can always
> refactor it into a more "diffuse" design like MDP/PPP.
>
> > I will need to put more effort into writing a message class to
> > organize the frames.
>
> There are many possibilities here!
>
> > From time to time I seriously wish ZMQ exposed more functions such as
> > getting all connected peers of a socket, or user side metadata of
> > messages etc.
>
> If it is consolation, I see many people sharing this comment.
>
> I do think that it is the right design choice for a libzmq socket to not
> provide peer tracking:
>
> It would require making both the libzmq API and ZMTP more complex and
> may negatively impact latency and throughput.  Maybe the tracking
> function could be made optional, but that still add software complexity.
>
> But, perhaps the most important reason is that any implementation of
> peer tracking makes an opinionated choice.  Eg, what does it mean for a
> peer to "go away"?  What does it mean when it reappears?  Is that a new
> peer or an old friend?  I think these kind of opinions are best
> expressed by the application itself.
>
> > Also after more reading of the code I totally agree with you on the
> > multi-part message. It does not seem to be a good idea to send a super
> > long multi-part message as it will block all other peers. Kind of
> > defeats the purpose.
>
> Well, just to be clear, neither multi-part nor largeness leads to any
> blocking of the application.  This is thanks to the per-peer
> output/input queues internal to the socket and to the transport between
> these queues being performed by ZeroMQ background "I/O threads".
>
> The thread-safety issue that multi-part messages pose strikes at the
> application-socket transfer boundary.  There, threads can race each
> other to call send() or recv() on the same socket and when calls from
> different sockets become interleaved the sequence of message parts would
> become mixed across the threads.
>
> Though, even if this particular kind of race is avoided we can not
> expect thread safety except for the sockets that specifically claim it
> (eg the "draft" sockets).
>
> -Brett.
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.zeromq.org/pipermail/zeromq-dev/attachments/20231029/3d8780ef/attachment.htm>