[zeromq-dev] Multicore Magic

[Martin Sustrik]

Hi Brian,

> Great article.  Our main usage case for 0MQ + Python is a parallel
> computing framework that we have been working on for years now.
> Originally we were targeting traditional Gb/Infiniband based clusters,
> but that entire landscape is really changing.  The new AMD 6100 CPUs
> that just came out are a huge step towards massively multicore.  You
> can now get a 48 core machine for not too much $.  But my word, what
> software can use 48 cores (other than using it as a virtualization
> platform)?  Other than Erlang, I can't think of anything better than
> 0MQ at this point.
> 
> Some other thoughts and comments:
> 
> * On a massively multicore machine, what transport layer makes to most
> sense?  IPC?  tcp:localhost?  What would be the fastest transport
> possible?

Obviously, inproc transport. However, in your case where Python is not 
able to run in parallel, it's presumably ipc.

IPC is faster than tcp:localhost on Linux, but exactly the same on 
Solaris. Other systems would have their own perf characteristics.

> * When using multiple massively multicore machines, which transport
> layer makes the most sense?  It would be nice if processes on
> different hosts could use tcp or infiniband, but processes on the same
> host could use something faster.

That's exactly the use case behind 0MQ being able to bind to multiple 
endpoints.

So you can for example bind a socket to TCP port _and_ to inproc 
endpoint. That way threads in the same process can connect via 
extra-fast inproc transport while processes on different boxes can use TCP.

> * Python still has the Global interpreter lock that prevents more than
> one thread from executing Python code at a time.  This means that
> thread based concurrency is not an option for Python.  IOW, we can't
> use multiple threads that talk using OMQ.  So, we have to use
> processes.  I don't like that, but maybe it is OK...

Well, it's deficiency of Python. Hopefuly Python developers would 
address the issue in the future.

Btw, maybe it would be worth poking them a little by asking about the 
behaviour on python dev mailing list...

Martin