[zeromq-dev] Non-contiguous message thoughts

Michael Santy michael.santy at dynetics.com
Sat Mar 6 19:49:16 CET 2010


> 
> I've been following different train of thought: When sending data like
> those in Michael's scenario, the splits are not random. Rather they
> mirror business logic of the application. For example, the structure of
> the message may look like this (matrixA, matrixB, matrixC).

Precisely -- we're just serializing data structures, sending, and then
reconstituting them on the other end.

> 
> In such a case - if we preserve the splits when message is delivered to
> the receiving end - it would be pretty easy to process the data. You'll
> simply process individual matrices, each one being guaranteed to reside
> in a contiguous memory region.
> 
> That brings me to the concept of "message groups" that's already
> provided by some messaging implementations. "Message group" is a group
> of messages that is guaranteed to be delivered in atomic fashion, i.e.
> you'll either get none of the messages in the group or you'll get all of
> them.
> 
> What about this:
> 
> zmq_send (s, msg1, ZMQ_GROUP);
> zmq_send (s, msg2, ZMQ_GROUP);
> zmq_send (s, msg3, ZMQ_GROUP);
> zmq_flush (s);

I thought about this a slightly different way, I was thinking that we could
create a zmq_msg_group_t type that was composed of a sequence of
zmq_msg_t's.  Its very analogous to what you propose, but your idea is
essentially a streaming message type.  I really like your idea better and
think it would be generally useful.

> 
> It's less trivial to implement (at very least it would require tweaking
> the wire protocol), however, the added value is much greater than in the
> simple iovec case.

Agreed, it certainly seems non-trivial, especially over some of the socket
types.  What did you have in mind for the API on the receiving side?  Would
the receiver have to know that it is receiving a message group, or would all
of the messages be delivered as one large message?  Or do you have another
idea?

Good discussion here.

Mike




More information about the zeromq-dev mailing list