[zeromq-dev] Contributing native InfiniBand/RDMA support to 0MQ

Gabriele Svelto gabriele.svelto at gmail.com
Fri Dec 16 09:20:40 CET 2011


 Hi Martin,

2011/12/16 Martin Lucina <martin at lucina.net>:
>
> Makes sense. Later we could look at using RDMA read/write for large
> messages.

That is a possibility, however it might require some fairly radical
changes to the API: to use RDMA operations memory needs to be pinned
first - which is an expensive operation - so generic user provided
data buffers won't work.

> Neat, the last one is basically PUB/SUB in hardware.

Yes, often with some logic on top, here's an interesting article if
you want to know more on this topic:

http://insidehpc.com/2011/03/01/special-feature-collective-performance-in-infiniband/

> You're right about SDP; I had a look at the state of distribution and
> kernel.org support and it does seem that ibverbs is the way to go.
>
> Also, AFAICT SDP is a kernel-space implementation so you don't get the
> possible benefits of a kernel bypass.
>
> Martin Sustrik and myself have a small test lab setup, so I will try and
> get ibverbs working there; have a couple of machines connected back to back
> with Mellanox ConnectX adapters. So far I've managed to get OpenSM, ibping,
> IPoIB working, but ibverbs is resisting (cannot find userspace driver...
> blah blah).

I can help with that, I've setup OFED on CentOS, Fedora, SUSE and
Gentoo so I've become accustomed with its quirks.

 Gabriele



More information about the zeromq-dev mailing list