[zeromq-dev] Load balancing REQ/REP sockets

[Martin Sustrik]

Jon Dyte wrote:
> Brian Granger wrote:
>> Hi,
>>
>> REQ/REP sockets have a load balancing feature.  If you have 1 master
>> with a REQ socket that binds:
>>
>>     ctx = zmq.Context()
>>     s = ctx.socket(zmq.REQ)
>>     s.bind('tcp://127.0.0.1:5555')
>>
>> And multiple workers that connect REP sockets to the REQ:
>>
>>     # Each worker does this.
>>     ctx = zmq.Context()
>>     s = ctx.socket(zmq.REP)
>>     s.connect('tcp://127.0.0.1:5555')
>>
>> When the master does a send, the requests will be load balanced to all
>> connected workers.  That works fine.
>>
>> But, a REQ socket cannot do another send until it gets back a reply
>> (by doing a recv).  So here is my question:
>>
>> How can you actually get load balancing in practice if the REQ socket
>> in the master cannot issue another send
>> (to a different worker) until first worker has replied.  Another way
>> of putting this is I don't see how to get the multiple
>> workers handling requests *simultaneously*.  Am I missing something or
>> is this by design.
>>
>> I can see how to accomplish this pattern with P2P sockets, but it
>> would require a much more complicated application logic to implement.
>>
>> thoughts?
>>
>> Cheers,
>>
>> Brian
>>
>>   
> you can get load balancing from multiple REQ sockets, but each 
> individual REQ socket
> effectively serialises each req and reply as far as I understand.
> 
> to do this intra-process you do something like see attached zmqserver.cpp
> which uses Xrep/xreq with a client like
> 
> // zmq
> #include <zmq.hpp>
> #include <unistd.h>
> #include <iostream>
> #include <stdio.h>
> 
> 
> int main(int argc , char*argv[])
> {
>  zmq::context_t ctx (1, 1);
> 
>  zmq::socket_t s (ctx, ZMQ_REQ);
> 
>  s.connect ("tcp://127.0.0.1:43210");
>  for (unsigned char i = 'A' ; i < 'Z' ; ++i)
>  {             
>    zmq::message_t request(1);
>      *((char*)request.data()) =  i;
>      printf("send %x\n", (int)     *((char*)request.data()));
>      s.send(request);
>        zmq::message_t response;
>      s.recv(&response);
>      printf("recv %x\n", (int)     *((char*)response.data()));
>      sleep(::getpid()%3);
>  }
>  return 0;
> 
> }
> 
> 
> you can also use the zmq_queue device which does the same thing but 
> doesnt use the inprocess endpoints.
> 
> thus you can have
> 
> REQ*<->SHARED_QUEUE(XREP,XREQ) <->REP*
> 
> * multiple of these, 1 shared queue.
> 
> caveat at the moment you have to do some special processing in the REP 
> socket.
> 
> Normally i have the shared queue do the binds for the xrep,xreq sockets 
> and then
> the workers(REP) and clients(REQ) connect to that.
> 
> There is a working zmq_queue device in the latest release.
> 
> does this make sense?

The whole XREQ/XREP business is still messy. If you absolutely need it 
at the moment you can use XREQ to connect and load-balance messages to 
mutliple REPs. That way you'll get what you need without a need to do 
any special processing on the REP side.

However, it's experimental functionality, so no warranty is given :)

Martin