[zeromq-dev] Usage case description and questions

[Martin Sustrik]

Brian,

First of all, thanks for your use case!

There are many different issues involved, so let's rather deal with it 
step by stem.

I believe the first thing to sort out is the routing model:

> * Dynamic load balancing.  In this scenario, tasks are sent to workers
> that are "least" busy.
> In english, this is "run this task ASAP anywhere you can."
> 
> * Worker specific scheduling.  By this, I mean we need the ability to
> track worker processes  by id/name/number and send particular tasks to
> that particular worker.
> In english this is "run this task only on this worker, even if it takes longer."
  Alright, that is a start.  Any thoughts, ideas or feedback would be
> greatly appreciated.

Let's have a look at the problem from the point of view of "services". 
As far as I understand what you have is "generic" service that will 
simply load-balance the request among the workers. Additionally you have 
a set of specific services (say service "A") that need to be handled by 
a specific worker.

Have a look at the attached diagram. REQ/REP model is used rather than 
PUB/SUB. Would it work that way?

(In case you would want several workers handling service A, one more 
shared queue can be added to the diagram.)

> As an aside, it looks like if we used zeromq, we could improve our
> latency by a factor of 10-100.  That would simply be amazing and would
> enable to scale up our system to very large cluster and
> supercomputers.

Yes, that was the idea behind 0MQ. The drawback is that development is 
much more labour intensive and adding new features without hurting 
performance is a lengthy process. Anyway, I worked on XREP and XREQ 
socket types over the weekend and it seems to be almost ready. These 
socket types would allow for routing replies back to original requester 
even over multiple network hops (see the shared queue component on the 
diagram).

Martin
-------------- next part --------------
A non-text attachment was scrubbed...
Name: services.png
Type: image/png
Size: 13299 bytes
Desc: not available
URL: <https://lists.zeromq.org/pipermail/zeromq-dev/attachments/20100215/e979a32d/attachment.png>