[zeromq-dev] Non-contiguous message thoughts

Dhammika Pathirana dhammika at gmail.com
Sat Mar 6 17:53:53 CET 2010

Hi Martin,

We can add scatter/gather IO API to zmq_msg. Proper readv/writev on
sockets will need some changes in encoder/decoder, but we can use
current encoder/decoder and read/write each iov entry separately.

typedef struct
    void *content;
    unsigned char shared;
    unsigned char vsm_size;
    unsigned char vsm_data [ZMQ_MAX_VSM_SIZE];
    struct iovec iov [ZMQ_MAX_IOV_SIZE];
} zmq_msg_t;

int zmq_msg_init_iov (zmq_msg_t *msg, int idx, void *data,
    size_t size, zmq_free_fn *ffn, void *hint);

On 3/5/10, Martin Sustrik <sustrik at 250bpm.com> wrote:
> Michael Santy wrote:
>  >
>  >> * You don't want to mess with lock that manage data across the IO/app
>  >> thread boundary.
>  >> * memcpy is not that slow, and by using it, you enable 0MQ to
>  >> deallocate the msg ASAP.
>  >>
>  >
>  > Thanks a bunch for the in-depth response.  Those are all good thoughts worth
>  > considering before moving any further on this idea. Just to give you a
>  > further idea of the problem, I'm passing ~200MB over DDR Infiniband every
>  > few seconds, and I just want to make sure that there's as little latency as
>  > possible in getting that data to the other process.  I'm not necessarily as
>  > concerned with lock contention on this, as the message rate is fairly low,
>  > so I think the deallocator approach will work well.  My issue is that the
>  > memory that I want to send it not contiguous in memory before creating a
>  > message.  Having to memcpy into a contiguous region of memory before send
>  > would add a little latency, but would also use memory bandwidth, which is a
>  > problem because I'm overlapping communication and computation.
> Btw, cheap and nasty solution would be to send you data as a sequence of
>  small messages (one message per allocated block).
>  This would of course break atomicity of message and mess with load
>  balancing (half of a message going to A, other half going to B).
>  However, it would at least allow you to test what the performance would
>  be if copying is avoided.
>  If you run such a test, please, post the results. It would be extremely
>  valuable.
>  Martin
> _______________________________________________
>  zeromq-dev mailing list
>  zeromq-dev at lists.zeromq.org
>  http://lists.zeromq.org/mailman/listinfo/zeromq-dev

More information about the zeromq-dev mailing list