[zeromq-dev] Non-contiguous message thoughts

Martin Sustrik sustrik at 250bpm.com
Sat Mar 6 19:39:15 CET 2010


> We can add scatter/gather IO API to zmq_msg. Proper readv/writev on
> sockets will need some changes in encoder/decoder, but we can use
> current encoder/decoder and read/write each iov entry separately.
> typedef struct
> {
>     void *content;
>     unsigned char shared;
>     unsigned char vsm_size;
>     unsigned char vsm_data [ZMQ_MAX_VSM_SIZE];
>     struct iovec iov [ZMQ_MAX_IOV_SIZE];
> } zmq_msg_t;
> int zmq_msg_init_iov (zmq_msg_t *msg, int idx, void *data,
>     size_t size, zmq_free_fn *ffn, void *hint);

Yes. It's doable, however, it would require quite an unpleasant change 
on the receiving end. Each receiving application would have to handle 
the case when the message is delivered in several non-contiguous chunks. 
Think of the added complexity needed to process data randomly split into 
several pieces!

I've been following different train of thought: When sending data like 
those in Michael's scenario, the splits are not random. Rather they 
mirror business logic of the application. For example, the structure of 
the message may look like this (matrixA, matrixB, matrixC).

In such a case - if we preserve the splits when message is delivered to 
the receiving end - it would be pretty easy to process the data. You'll 
simply process individual matrices, each one being guaranteed to reside 
in a contiguous memory region.

That brings me to the concept of "message groups" that's already 
provided by some messaging implementations. "Message group" is a group 
of messages that is guaranteed to be delivered in atomic fashion, i.e. 
you'll either get none of the messages in the group or you'll get all of 

What about this:

zmq_send (s, msg1, ZMQ_GROUP);
zmq_send (s, msg2, ZMQ_GROUP);
zmq_send (s, msg3, ZMQ_GROUP);
zmq_flush (s);

It's less trivial to implement (at very least it would require tweaking 
the wire protocol), however, the added value is much greater than in the 
simple iovec case.


More information about the zeromq-dev mailing list