[zeromq-dev] Multicore Magic

Brian Granger ellisonbg at gmail.com
Fri Apr 23 16:48:10 CEST 2010


Great article.  Our main usage case for 0MQ + Python is a parallel
computing framework that we have been working on for years now.
Originally we were targeting traditional Gb/Infiniband based clusters,
but that entire landscape is really changing.  The new AMD 6100 CPUs
that just came out are a huge step towards massively multicore.  You
can now get a 48 core machine for not too much $.  But my word, what
software can use 48 cores (other than using it as a virtualization
platform)?  Other than Erlang, I can't think of anything better than
0MQ at this point.

Some other thoughts and comments:

* On a massively multicore machine, what transport layer makes to most
sense?  IPC?  tcp:localhost?  What would be the fastest transport
* When using multiple massively multicore machines, which transport
layer makes the most sense?  It would be nice if processes on
different hosts could use tcp or infiniband, but processes on the same
host could use something faster.
* Python still has the Global interpreter lock that prevents more than
one thread from executing Python code at a time.  This means that
thread based concurrency is not an option for Python.  IOW, we can't
use multiple threads that talk using OMQ.  So, we have to use
processes.  I don't like that, but maybe it is OK...
* What fun!



On Fri, Apr 23, 2010 at 7:02 AM, Pieter Hintjens <ph at imatix.com> wrote:
> Hi All,
> We've published an article on 0MQ as the solution to multithreading woes, here:
> http://www.zeromq.org/blog:multithreading-magic
> (It's not really a blog but this allows for comments.)
> Cheers,
> --
> Pieter
> _______________________________________________
> zeromq-dev mailing list
> zeromq-dev at lists.zeromq.org
> http://lists.zeromq.org/mailman/listinfo/zeromq-dev

Brian E. Granger, Ph.D.
Assistant Professor of Physics
Cal Poly State University, San Luis Obispo
bgranger at calpoly.edu
ellisonbg at gmail.com

More information about the zeromq-dev mailing list