[zeromq-dev] 0MQ/2.0-alpha3 available

Martin Sustrik sustrik at fastmq.com
Tue Sep 29 11:10:38 CEST 2009


Hi Aamir,

> The request / reply socket is described in zmq.h as follows:
> 
> //  Socket to send requests and receive replies. Requests are
> //  load-balanced among all the peers. This socket type allows
> //  only an alternated sequence of send's and recv's
> #define ZMQ_REQ 3
> 
> This is the "synchronous RPC" case that we discussed earlier. I'm having 
> some difficultly understanding how the load-balancing can work with the 
> restriction that only alternated sequences of sends and receives are 
> allowed. 

Exactly. This is the simplest possible req/rep scenario (RPC). The goal 
is to support more complex scenarios as well, presumably using different 
socket types. Everyone is welcome to present their use cases and propose 
semantics for new socket types to support those scenarios!

> Let's say I have 100 cores running, and I want to distribute 100 
> requests (and then collect the corresponding 100 replies). If I use 
> ZMQ_REQ then I will have to wait for the first request to be completed 
> before the second request can be sent. Wouldn't this mean that only 1 
> core is ever working at any given time? If this is true, then it defeats 
> the purpose of load-balancing the requests and raises a serious question 
> about the usefulness of ZMQ_REQ and ZMQ_REP? 

Yes, in the above scenario it does defeat the purpose. The 
load-balancing mechanism makes sense only if there are many requesters 
and limited number of repliers (services). The way it works now the 
requests are split randomly between available repliers resulting in a 
kind of stochastic load-balancing.

> Using 0MQ 1.0, I implemented an asynchronous request / reply system 
> based on the load-balancing exchange object. It allows me to send 100 
> requests to a load-balancing exchange from a sender thread, without 
> waiting for any replies. A separate receiver thread listens for replies. 
> As a result I am able to keep all 100 cores working simultaneously. This 
> design works "well enough' as long as there is only ever one requester 
> application running (otherwise the repliers would not know who to send 
> the reply back to). There is an added difficulty that replies will not 
> arrive in the same order as the requests were sent, so we need to check 
> the message contents to figure out which reply corresponds to which 
> request (and whether or not all requests have been replied to once and 
> only once).  With 0MQ 2.0, there is no concept of a "load-balancing 
> exchange." Instead there is a ZMQ_REQ socket, but the ZMQ_REQ is 
> fundamentally different as far as I can tell. So, as it is, I would not 
> be able to migrate my code to 0MQ 2.0.
> I think the simplest thing to do is to just have a load-balancing 
> version of ZMQ_PUB that only supports sends (which is very similar to 
> the load-balancing exchange concept). An added layer of complexity would 
> be for the 0MQ infrastructure to support routing asynchronous replies 
> based on some kind of "reply-to" field contained in the corresponding 
> request message (I think this is a network-layer issue and would ideally 
> be supported by 0MQ). A further layer of complexity is to use the 
> message contents to re-order the replies so that they are in the same 
> order in which the corresponding requests were sent (this seems to be 
> potentially an application-specific issue that goes beyond the scope of 
> a light-weight messaging library).

Right. Let's design new socket types to support your use case. As far as 
I understand it's pretty similar to the "butterfly" scenario described here:

http://www.zeromq.org/tutorials:butterfly

In any case, you've pointed out the problems associated with those 
scenarios correctly. Following issues should be decided upon:

1. Do we want to route the replies back to the sender or do we just want 
to propagate them further on? In former case some kind of route-back 
mechanism mas to be involved.

2. Do we want to reorder replies so that sequence of requests is 
matched? If so, message numbering should be added.

My intuition is that there we are probably having two distinct use cases 
here, one for streamlining messages forth, the other one for routing the 
replies back to the requester ("anyc requester").

Martin



More information about the zeromq-dev mailing list