[zeromq-dev] How zeromq handles slow consumers with PUB/SUB mode
sustrik at 250bpm.com
Tue Mar 23 16:33:18 CET 2010
> I'm planning to write a prototype using zeromq. I see in the
> documentation that zeromq provides
> a publisher/subscriber mode. I want to use this mode in my application,
> but i'm asking how zeromq handles
> slow consumer.
> Is there some documentation about this point ?
The documentation is lacking in this area :(
However, the principle is pretty simple: You can think of the network as
series of tubes. Each tube has some max capacity. For example, when
communicating using 2 0MQ sockets there's senders 0MQ queue, kernel
SNDBUF, queue in the NIC, queue in the switch, queue in the receiver's
NIC, kernel RCVBUF, receiver's 0MQ queue...
When consumer is slow these queues start growing. The process starts at
the place of the bottleneck then propagates up back to the sender. Once
all the queues are full, meaning that sender cannot send another message
without exceeding the limits, some measure must be taken.
The exact measure depends on the messaging pattern. For example for
REQ/REP send function blocks in such a situation. For PUB/SUB the
message is accepted and dropped.
While the latter looks scary, it makes perfect sense because blocking
would cause whole system to stop. Even innocent subscribers would not
If you want to reasonably guarantee the delivery, the only way is to
make the queues very large. We've seen queues set to max. 200GB to be
able to hold whole trading day of market data.
More information about the zeromq-dev