[zeromq-dev] Parallel sort

[Oliver Smith]

On 8/2/2010 8:07 PM, Pieter Hintjens wrote:
>> However: if you introduce the overhead of having to create threads on
>> demand, rather than being able to use a pool of pre-existing workers,
>> you'd probably create a situation where - for purely local
>> implementations - OpenMP's "#pragma omp task" or TBB's "TBB::Pipeline"
>> system would be better (tbb in-particular if you are using Intel hardware :)
>>      
> This is why I wonder about the size of your sort set.  A few thousand
> threads would be fine.  Millions would be weird IMO.  There are
> certainly ways to map a very large virtual tree onto a smaller number
> of real worker threads. It just requires some passing of state up and
> down.  Grab state, decide whether to split or execute, return results,
> repeat.  Would work with stateless workers and all state concentrated
> in one dispatcher.
>    
Ah - I don't actually have a work load to sort right now, rather I'm 
still working on the whole ZeroMQ vs OpenMP/TBB thing. Parallel sort is 
typically the first thing these libraries demonstrate, leading on to how 
to handle recursive parallelism etc.

- Oliver