[zeromq-dev] Load balancing REQ/REP sockets
sustrik at 250bpm.com
Thu Mar 18 13:45:14 CET 2010
>> Yes. The meachism intended for this kind of thing is "queue limits"
>> mechanism recently ported from 0MQ/1.0.
> This looks like it may do what I need, but I want to understand this better.
>> You can set ZMQ_HWM socket option on a socket, specifying max number of
>> messages it can hold. Once the limit is reached for specific connection it
>> stops accepting new messages and load balancing skips it in the round-robin.
> - Do all socket types have both an incoming+outgoing queue?
Those socket types that can be used for sending have outgoing queue(s).
Those socket types that can be used for receiving have incoming queue(s).
> - Does ZMQ_HWM affect the socket's incoming queue or outgoing queue or both?
Hm. It's both. But I would say you are right and the two limits should
be separated same way as SO_SENDBUF is separated from SO_RECVBUF.
> - In the XREQ + multiple REP socket situation which side do I set HWM on?
There are queues on both sides. Queue on the sender side stores messages
that have been sent from the API and not yet passed to the network.
Queue on the receiving side stores messages that have been read from the
wire but not yet recvd by the application.
> I have set HWM and LWM to 2 in the XREQ + multiple REP situation. The
> XREQ can keep sending messages, but where do they go? It sounds like
> the XREQ holds the messages in its outgoing queue?
I am not sure - are you speaking of the case where there's no REP
connected to your XREQ?
>> However, the number of requests "on the fly" can be pretty high. There can
>> be messages in the sender's queue, in secder's TCP tx window, on the network
>> (queue on a switch etc.), in receiver's rx window and in receiver's queue.
> This makes sense. But where in all of this is the HWM enforced?
> Receivers queue?
Each limit is enforced on particular point of the stack. Senders HWM is
enforced on 0MQ level, TCP tx window limit is enforced on TCP level,
switch enforces max packet queue size etc.
>> What you are speaking about is making it work in lock-step fashion, i.e. no
>> new message is sent until the reply is received. This should work on a REQ
>> socket. If it does not, it should be fixed.
> Yes, I think the standard REQ is working as expected.
>> With XREQ socket the situation is a bit more complex. The user can send many
>> requests without waiting for a reply. Some of the requests are routed
>> through a single connection. At the end of the connection there may be
>> another load-balancer, such as zmq_queue. If it worked in lock-step fashion.
>> The second load-balancer would be completely useless as there would be at
>> most one request processed at a time.
> Yes, I can see that what I propose doesn't fit well with this
> situation. Let me see if I can understand HWM and get it working for
> my situation.
More information about the zeromq-dev