[zeromq-dev] Multi-part messages

Brian Granger ellisonbg at gmail.com
Sat Mar 27 18:46:43 CET 2010


Martin,

> No API for it now. I have no sane idea how the API should look like :(
>
> As an workaround you can check the flag by hand:
>
> zmq_msg_t msg;
> ...
> int tbc = msg.flags & ZMQ_MSG_TBC ? 1 : 0;

I guess this is the API then :)  I will think about how to represent
this in Python.

>> I guess this means the routing string of XREP will be handled in a
>> multipart.  What will the algorithm be to cleanly separate the 0MQ
>> usage of multipart from the user's usage of it?  Will 0MQ internally
>> add and remove message parts so that the user only sees the
>> application message parts at the endpoints?
>
> Yes. 0MQ will do that for you. In multi-hop req/rep scenario each
> intermediate node will add it's ID to the request. Terminal REP node would
> copy the stack of IDs to the reply and send it back. Each intermediate node
> would trim one ID from the reply.

Fantastic.

>> Another question: how do multipart message interact with topic
>> filtering.  Is topic filtering done on each message in a group
>> separately?  On the entire group?  Is the topic the first message in
>> the group?
>
> Only the first part is used for topic matching. Not doing so would result in
> multi-part message not being atomic.

Great, this is exactly what we need!

>> One thing we are running into with Python is that if we recv a large
>> message that has a topic and routing info (XREP) we get the result as
>> a single large message.  Because Python strings are immutable, we have
>> to make expensive copies to extract the message.  It would be great if
>> we could use multipart to get the tag, routing info and message in
>> different recv's.
>
> Yes, exactly. Multi-part messages should solve the zero-copy problems for
> both 0MQ itself and user applications alike.

Again, great.

>>> The patch is quite complex and thus it would benefit from testing.
>>
>> I will write a test suite in pyzmq for this.
>
> Great!
>
>>> Also note that XREQ and XREP sockets are rendered unfunctional by the
>>> patch and will be rewritten shortly.
>>
>> We are using XREQ and XREP extensively in a real app now, so this will
>> slow us a bit.  Other than handling the routing info as using message
>> parts, will the functionality of XREP/XREQ change?
>
> No it won't except for REP socket where user currently has to manage the
> routing prefix by hand. It will be handled transparently for the user. And
> it's next on my todo list, so hopefully you can start using it shortly.

OK, this sounds good.  I recently added a simple API to pyzmq for
handling the routing prefix by hand, but it would be nice to not have
to do that.

Cheers,

Brian

> Martin
>



-- 
Brian E. Granger, Ph.D.
Assistant Professor of Physics
Cal Poly State University, San Luis Obispo
bgranger at calpoly.edu
ellisonbg at gmail.com



More information about the zeromq-dev mailing list