[zeromq-dev] Contributing native InfiniBand/RDMA support to 0MQ

[Martin Sustrik]

Hi Gabriele,

> I was considering contributing native support for InfiniBand and other
> RDMA-enabled technologies to 0MQ and wished to know if there was some
> interest in it first. I have played a little bit with the project and
> found it very interesting, and I think that native support for
> RDMA-enabled technologies would be a nice addition to it.

I think it would be a killer feature for HPC and financial services 
users. So, definitely, go for it!

> - I would introduce a new transport using the Reliable Connection (RC)
> protocol, this is similar to TCP in that it guarantees lossless,
> ordered message delivery though contrary to TCP it delivers messages
> as datagrams

Right. And given the transport is lossless, it should be easy to 
fragment messages into datagrams.

> - Connection management would be done using the RDMA connection
> manager library, this has semantics and an interface which resembles
> very closely TCP sockets and provides file descriptors for
> notification; establishing connections wouldn't look too different
> from what happens in tcp_listener_t/tcp_connecter_t

Yes.

> - Data transfer would be done with raw verbs calls, this would make
> use of a custom sender/receiver object, something akin to
> stream_engine_t. The ibverbs library supports file descriptors for
> signaling data transfer events so this should also fit well within the
> 0MQ model

Exactly.

> - The code would mainly go into separate files plus some changes to
> socket_base.cpp and session_base.cpp to support the new transport,
> relevant options and checks would be added to the autoconf
> infrastructure

Yes.

> - The project would be pretty much Linux-only though Windows support
> might be introduced at a later stage, FreeBSD support might come along
> when they finish porting the OFED stack to their kernel

Yes. This can be done on one-by-one basis. We have OS checking macros in 
0MQ so it should be easy. Also, it would be nice to have --with-ib or 
--with-rdma  option to configure to switch the compiling of this 
component on. (Same as --with-pgm we are using to switch PGM support on.)

> - Looking at the contribution page it seems to me that the best model
> for such a project would be to fork the mainline on GitHub and ask for
> pull requests if/when there will be interest in integrating it in the
> main codebase; I am also under the impression that it would not be too
> hard to backport the changes to version 2.x at a later stage

Yes. Use mainline for new development. The code was much simplified 
since 2.x times, so it'll be much easier for you.

> So my question is: before I start working on it, is there any interest
> for this? I would do it mostly for fun and because I find RDMA
> networking technologies very powerful tools that are unfortunately
> fairly hard to use for the non-initiated and I feel that adding
> support in 0MQ would make them more accessible to a wider public.

Yes. I think that's exactly the point. Using IB is not easy (unless you 
run IPoIB on top of it) and this kind of project can possibly make it a 
viable option for those without the extensive experience with RDMA.

Martin