[zeromq-dev] Urgent! Where is the link between ZeroMq and IpoIB

Jim Garlick garlick at llnl.gov
Wed May 7 17:37:08 CEST 2014


I am not too familiar with zeromq internals, but I think the right approach
would be to create a new transport (e.g. rdma://) and interface directly
with the user space InfiniBand stack, e.g. ibverbs or SDP or something.
See earlier thread on zeromq-dev as I believe different approaches
were discussed.

Hooking into MPI seems awkward given that MPI could be considered a
higher level library than zeromq, requires support from a launcher/resource
manager, etc.. 

Jim

On Wed, May 07, 2014 at 11:49:25AM +0200, Krishna Narasimhan wrote:
> Hi Jim,
>     Thanks for the response. So then I would like to ask you one follow up
> question : (hopefully the last since you have been surprisingly patient)
> 
> Lets assume I write a wrapper that directs to libraries like OpenMPI (with
> direct RDMA support) or the old calls in Zeromq depending on the hardware
> available, where do you think I should start looking. I mean,
> 
> 1) Is it possible to do such a thing where I pick out calls to the IP
> stack(indirectly IPoIB) and replace them with the wrapper?
> 
> 2) If so, which are the calls I should start looking at and how do I
> identify them in the Zeromq code?
> 
> 
> 
> On Tue, May 6, 2014 at 6:51 PM, Jim Garlick <garlick at llnl.gov> wrote:
> 
> > I'm speaking of Linux (RHEL 6.x) where IPoIB presents a regular 'ib0'
> > network interface and is fully integrated with the kernel IP stack.
> > Zeromq thus doesn't need (and doesn't have) special linkage to the
> > user space InfinBand libraries.
> >
> > Jim
> >
> > On Tue, May 06, 2014 at 04:16:15PM +0200, Krishna Narasimhan wrote:
> > > Hi Jim,
> > >     Thanks a lot for the response. Really glad you took time out to read
> > my
> > > issue completely.
> > >
> > > It is really interesting the links you have provided. Good to know that
> > > there was prior attempts at this.
> > >
> > > Btw, I dont understand still how ZeroMq is linked to IpOIb. Do you mean
> > to
> > > say that ZeroMq code never changed, but the underlying Socket mechanism
> > > takes care of the decision of normal or (IpoIB sockets) based on the
> > > request and hardware present?
> > >
> > > I saw some codes like
> > >
> > >  fd_t s = socket (domain_, type_, protocol_);
> > >
> > >
> > > Is this where the dynamic choice of using IpoIb happens?
> > >
> > > I am not sure I explained my question properly.
> > >
> > > I understand that IpoIb codes are available in a linux driver code , so I
> > > was looking for references to IpoIb.h
> > >
> > > If all my above explanation turned out to be confusing , here goes my
> > last
> > > attempt :
> > >
> > > How are Zeromw code and IpoIb.h linked?
> > >
> > >
> > > On Mon, May 5, 2014 at 10:53 PM, Jim Garlick <garlick at llnl.gov> wrote:
> > >
> > > > Hi Krishna,
> > > >
> > > > As far as I know ZeroMQ can only use IPoIB via normal IP sockets.
> > > > Looks like the idea of an RDMA zeromq transport came up in 2011,
> > > > but went nowhere?
> > > >
> > > > http://lists.zeromq.org/pipermail/zeromq-dev/2011-December/014784.html
> > > >
> > > > Another interesting example of adding RDMA support to a project with
> > > > pluggable transports is net/9p (transport for fs/v9fs) in the Linux
> > kernel.
> > > > It might be useful to look at that code and follow its history to
> > > > understand
> > > > the problem better.
> > > >
> > > > http://lwn.net/Articles/301896/
> > > >
> > > > As I recall the orig code in 2008, though functional, didn't deliver a
> > big
> > > > performance win over TCP (for that stack) and introduced new failure
> > modes,
> > > > making it unattractive and frustrating for casual users.  It languished
> > > > for years in a semi-broken state until people at CEA/Bull began working
> > > > on it again in 2012-13.  I'm guessing the second order issues CEA/Bull
> > > > dealt with would be instructive.  Also their Mooshika RDMA abstraction,
> > > > which I think allows for a wireshark tap and testing w/o Infiniband
> > card
> > > > could be interesting.
> > > >
> > > > https://github.com/martinetd/mooshika
> > > >
> > > > Jim
> > > >
> > > > On Mon, May 05, 2014 at 07:38:05PM +0200, Krishna Narasimhan wrote:
> > > > > Hello,
> > > > >         I am a researcher at Frankfurt University and currently
> > working
> > > > on
> > > > > contributing to folks who use ZeroMq at Gsi , Darmstadt. Can anyone
> > guide
> > > > > me on which part of the code links ZeroMq to IPoIB. Because it doesnt
> > > > seem
> > > > > to be linked at code level. Does ZeroMq call some wrapper which
> > directs
> > > > to
> > > > > IPoIb or normal IP?
> > > > >
> > > > > Any help here would be appreciated. this is rather urgent.
> > > > >
> > > > > On the other hand, I am also looking to contribute to including MPI
> > > > (direct
> > > > > RDMA) access for ZeroMq based applications, so anyone with this
> > knowledge
> > > > > will also be helpful. But am not sure this belongs to this thread.
> > > > >
> > > > > Thanks a lot.
> > > > >
> > > > > Regards
> > > > >        Krishna
> > > >
> > > > > _______________________________________________
> > > > > zeromq-dev mailing list
> > > > > zeromq-dev at lists.zeromq.org
> > > > > http://lists.zeromq.org/mailman/listinfo/zeromq-dev
> > > >
> > > > _______________________________________________
> > > > zeromq-dev mailing list
> > > > zeromq-dev at lists.zeromq.org
> > > > http://lists.zeromq.org/mailman/listinfo/zeromq-dev
> > > >
> > >
> > >
> > >
> > > --
> > > -----------------------------------------------------
> > > I dare do all that may become a man; Who dares do more, is none -
> > Macbeth,
> > > twelfh night!
> > > Regards
> > >        Krishna
> >
> 
> 
> 
> -- 
> -----------------------------------------------------
> I dare do all that may become a man; Who dares do more, is none - Macbeth,
> twelfh night!
> Regards
>        Krishna



More information about the zeromq-dev mailing list