[zeromq-dev] Load balancing and fault tolerance

Matt Weinstein matt_weinstein at yahoo.com
Tue Jul 13 15:06:26 CEST 2010


(( Sorry, I was typing on an iPad. In a moving car. Heading for a  
train. :-) ))

Originally, I figured it would be easy to just use an XREP directly,  
and interpret and dispatch, but that's no fun.

Another way to do it might be to cheat, and use two XREPs back to  
back, i.e. SERVICE THREADS emit a REQUEST when they start up.  This  
identifies them to your specially crafted DEVICE, which places them in  
a FIFO queue.

When a real CLIENT REQUEST comes in from the other side, you pull the  
next available SERVICE THREAD uuid off the FIFO, and feed the CLIENT  
REQUEST to the SERVICE THREAD (as a response!).  Now you know how is  
handling the request, this information can be treasured away.

When the response comes back from the SERVICE THREAD, the response is  
forwarded to the CLIENT, and the SERVICE THREAD's UUID is put back in  
the FIFO.

If there is a TIMEOUT (use the 3rd socket approach, per prior email  
threads) another SERVICE THREAD is pulled off the FIFO, and the CLIENT  
REQUEST is re-sent to the new SERVICE THREAD.  State is recorded to  
make sure the request is only responded to once.

You can only poll on the inbound socket (CLIENT REQUEST side) if you  
actually have SERVICE THREADS in the FIFO.  Fortunately, you are  
single threaded, so you will hear about that from the other socket  
shortly :-)

If you need some type of buffering, you can add another buffering  
DEVICE (ZMQ_QUEUE) cross connected with your special device and two  
ZMQ_STREAMs.

As an added plus, I believe the request UIDs can be assigned at this  
layer, so you don't even need to track them at the CLIENT side.

I believe this maintains all of the socket state correctly, but this  
should be walked through carefully.

That's the sketch anyway.

If this works, I'll probably use it too :-)

Comments welcome ...

Best,

Matt


On Jul 13, 2010, at 7:28 AM, Brian Granger wrote:

> On Tue, Jul 13, 2010 at 3:39 AM, Matt Weinstein
> <matt_weinstein at yahoo.com> wrote:
>> The XREQ / XREP protocol carries the GUID of the requester in the  
>> header
>> part.
>>
>> client -->  [ REQ ] --- [ XREP ]  === (GUID of client available  
>> here) ==  [
>> XREQ ] [ REP ]  your workers here
>>
>> I'm using it to track timeouts.
>>
>> Does that help?
>
> Not really.  We need the ability to track the location of each message
> in the worker pool.  The GUIDs track which client submitted the
> message.
>
> Brian
>
>> On Jul 13, 2010, at 3:41 AM, Brian Granger wrote:
>>
>>> Hi,
>>>
>>> One of the core parts of a system I am building with zeromq is a  
>>> task
>>> distribution system.  The system has 1 master process and N workers.
>>> I would like to use XREP/XREQ sockets for this, but am running into
>>> two problems:
>>>
>>> 1.  The load balancing algorithm (round robin) of the XREQ socket is
>>> not efficient for certain types of loads.  I think the recent
>>> discussions on other load balancing algorithms could help in this
>>> regard.
>>>
>>> 2.  Fault tolerance.  If a worker goes down, I need to know about it
>>> and be able to requeue any tasks that went down with the worker.  To
>>> monitor the health of workers, we have implemented a heartbeat
>>> mechanism that works great.  BUT, with the current interface, I  
>>> don't
>>> have any way of discovering which tasks were on the worker that went
>>> down.  This is because the routing information (which client gets a
>>> message) is not exposed in the API.
>>>
>>> Because of this second limitation, we are having to look at crazy
>>> things like using a PAIR socket for each worker.  This is quite a  
>>> pain
>>> because I can't use the 0MQ load balancing and I have to manage  
>>> all of
>>> those PAIR sockets (1 for each worker).  With lots of workers this
>>> doesn't scale very well.
>>>
>>> Any thought on how to make XREQ/XREP more fault tolerant?
>>>
>>> Cheers,
>>>
>>> Brian
>>>
>>> --
>>> Brian E. Granger, Ph.D.
>>> Assistant Professor of Physics
>>> Cal Poly State University, San Luis Obispo
>>> bgranger at calpoly.edu
>>> ellisonbg at gmail.com
>>> _______________________________________________
>>> zeromq-dev mailing list
>>> zeromq-dev at lists.zeromq.org
>>> http://lists.zeromq.org/mailman/listinfo/zeromq-dev
>>
>>
>
>
>
> -- 
> Brian E. Granger, Ph.D.
> Assistant Professor of Physics
> Cal Poly State University, San Luis Obispo
> bgranger at calpoly.edu
> ellisonbg at gmail.com




More information about the zeromq-dev mailing list