[zeromq-dev] Multicore Magic

[Brian Granger]

On Mon, Apr 26, 2010 at 11:11 AM, Pieter Hintjens <pieterh at gmail.com> wrote:
> On Mon, Apr 26, 2010 at 7:43 PM, Brian Granger <ellisonbg at gmail.com> wrote:
>
>> I agree that for now the GIL makes it easy to integrate with
>> C-extensions.  BUT, the GIL is killing Python's chances to be a useful
>> language for multicore.  Period.
>
> If you could start a Python process instance for each process, with
> 0MQ you could do multithreading nicely, as explained in the article.
> I presume one could then use inproc transport.


You can *definitely* start multiple Python processes on a host and use
the ipc transport.  But as I understand it, the inproc transport only
works between *threads* in a single process.  Thus, with Python if you
want parallelism you need to use *processes* and the slower ipc/tcp
transports.  I could be misunderstanding what the inproc transport
does, but I think this is how it works.

> What's the feasibility of running multiple independent Pythons, one
> per core?  I assume memory's not a major issue any more.

As processes this works fine, I just think we have to use the slower
ipc/tcp transports.  I will try to look at this soon though.

Brian

> -Pieter
>



-- 
Brian E. Granger, Ph.D.
Assistant Professor of Physics
Cal Poly State University, San Luis Obispo
bgranger at calpoly.edu
ellisonbg at gmail.com