[zeromq-dev] Migrating IPC-based Pub/Sub System To ZMQ

[Pieter Hintjens]

On Mon, Aug 16, 2010 at 7:29 PM, Santy, Michael
<Michael.Santy at dynetics.com> wrote:
>> PUSH/PULL (aka DOWNSTREAM/UPSTREAM) will load-balance requests among
>> processors. They are designed precisely for this use case.
>
> Because of the variability in computation time on the worker nodes we don't
> use round-robin, but instead use least-recently-used (LRU) scheduling.  When
> done processing a worker reports that it is ready, so the scheduling logic
> must be in the application.  If the scheduling was round-robin, the
> DOWNSTREAM/UPSTREAM would be a perfect fit.  It sounds like I'll need to
> come up with some application-level workaround for this use-case.

Indeed, round-robin is not perfect for all types of workload
distribution.  You can consider using a reverse request-reply
mechanism where the workers signal 'ready' by sending a request.

You'd want two threads in the collector.  First thread collects data,
prepares workloads, sends to second thread.  Second thread listens on
REP socket, pops workloads and sends them as replies.

Let me know if you want a worked example, I'm thinking this could make
a nice example for the tutorial of inter-thread signalling.

-
Pieter Hintjens
iMatix - www.imatix.com