[zeromq-dev] Non-contiguous message thoughts
Dhammika Pathirana
dhammika at gmail.com
Sat Mar 6 17:53:53 CET 2010
Hi Martin,
We can add scatter/gather IO API to zmq_msg. Proper readv/writev on
sockets will need some changes in encoder/decoder, but we can use
current encoder/decoder and read/write each iov entry separately.
typedef struct
{
void *content;
unsigned char shared;
unsigned char vsm_size;
unsigned char vsm_data [ZMQ_MAX_VSM_SIZE];
struct iovec iov [ZMQ_MAX_IOV_SIZE];
} zmq_msg_t;
int zmq_msg_init_iov (zmq_msg_t *msg, int idx, void *data,
size_t size, zmq_free_fn *ffn, void *hint);
On 3/5/10, Martin Sustrik <sustrik at 250bpm.com> wrote:
> Michael Santy wrote:
> >
> >> * You don't want to mess with lock that manage data across the IO/app
> >> thread boundary.
> >> * memcpy is not that slow, and by using it, you enable 0MQ to
> >> deallocate the msg ASAP.
> >>
> >
> > Thanks a bunch for the in-depth response. Those are all good thoughts worth
> > considering before moving any further on this idea. Just to give you a
> > further idea of the problem, I'm passing ~200MB over DDR Infiniband every
> > few seconds, and I just want to make sure that there's as little latency as
> > possible in getting that data to the other process. I'm not necessarily as
> > concerned with lock contention on this, as the message rate is fairly low,
> > so I think the deallocator approach will work well. My issue is that the
> > memory that I want to send it not contiguous in memory before creating a
> > message. Having to memcpy into a contiguous region of memory before send
> > would add a little latency, but would also use memory bandwidth, which is a
> > problem because I'm overlapping communication and computation.
>
>
> Btw, cheap and nasty solution would be to send you data as a sequence of
> small messages (one message per allocated block).
>
> This would of course break atomicity of message and mess with load
> balancing (half of a message going to A, other half going to B).
>
> However, it would at least allow you to test what the performance would
> be if copying is avoided.
>
> If you run such a test, please, post the results. It would be extremely
> valuable.
>
>
> Martin
>
> _______________________________________________
> zeromq-dev mailing list
> zeromq-dev at lists.zeromq.org
> http://lists.zeromq.org/mailman/listinfo/zeromq-dev
>
More information about the zeromq-dev
mailing list