[zeromq-dev] Parallel sort

[Pieter Hintjens]

On Tue, Aug 3, 2010 at 2:45 AM, Oliver Smith <oliver at kfs.org> wrote:

> However: if you introduce the overhead of having to create threads on
> demand, rather than being able to use a pool of pre-existing workers,
> you'd probably create a situation where - for purely local
> implementations - OpenMP's "#pragma omp task" or TBB's "TBB::Pipeline"
> system would be better (tbb in-particular if you are using Intel hardware :)

This is why I wonder about the size of your sort set.  A few thousand
threads would be fine.  Millions would be weird IMO.  There are
certainly ways to map a very large virtual tree onto a smaller number
of real worker threads. It just requires some passing of state up and
down.  Grab state, decide whether to split or execute, return results,
repeat.  Would work with stateless workers and all state concentrated
in one dispatcher.

-Pieter