[zeromq-dev] Non-contiguous message thoughts

[Martin Sustrik]

Michael Santy wrote:
>   
>> * You don't want to mess with lock that manage data across the IO/app
>> thread boundary.
>> * memcpy is not that slow, and by using it, you enable 0MQ to
>> deallocate the msg ASAP.
>>
> 
> Thanks a bunch for the in-depth response.  Those are all good thoughts worth
> considering before moving any further on this idea. Just to give you a
> further idea of the problem, I'm passing ~200MB over DDR Infiniband every
> few seconds, and I just want to make sure that there's as little latency as
> possible in getting that data to the other process.  I'm not necessarily as
> concerned with lock contention on this, as the message rate is fairly low,
> so I think the deallocator approach will work well.  My issue is that the
> memory that I want to send it not contiguous in memory before creating a
> message.  Having to memcpy into a contiguous region of memory before send
> would add a little latency, but would also use memory bandwidth, which is a
> problem because I'm overlapping communication and computation.

Btw, cheap and nasty solution would be to send you data as a sequence of 
small messages (one message per allocated block).

This would of course break atomicity of message and mess with load 
balancing (half of a message going to A, other half going to B).

However, it would at least allow you to test what the performance would 
be if copying is avoided.

If you run such a test, please, post the results. It would be extremely 
valuable.

Martin