[zeromq-dev] How zeromq handles slow consumers with PUB/SUB mode
joel.vennin at gmail.com
Tue Mar 23 18:47:21 CET 2010
Thank you for these useful information !
On Tue, Mar 23, 2010 at 4:33 PM, Martin Sustrik <sustrik at 250bpm.com> wrote:
> Hi Joel,
> I'm planning to write a prototype using zeromq. I see in the documentation
>> that zeromq provides a publisher/subscriber mode. I want to use this mode in
>> my application, but i'm asking how zeromq handles
>> slow consumer.
>> Is there some documentation about this point ?
> The documentation is lacking in this area :(
> However, the principle is pretty simple: You can think of the network as
> series of tubes. Each tube has some max capacity. For example, when
> communicating using 2 0MQ sockets there's senders 0MQ queue, kernel SNDBUF,
> queue in the NIC, queue in the switch, queue in the receiver's NIC, kernel
> RCVBUF, receiver's 0MQ queue...
> When consumer is slow these queues start growing. The process starts at the
> place of the bottleneck then propagates up back to the sender. Once all the
> queues are full, meaning that sender cannot send another message without
> exceeding the limits, some measure must be taken.
> The exact measure depends on the messaging pattern. For example for REQ/REP
> send function blocks in such a situation. For PUB/SUB the message is
> accepted and dropped.
> While the latter looks scary, it makes perfect sense because blocking would
> cause whole system to stop. Even innocent subscribers would not get
> If you want to reasonably guarantee the delivery, the only way is to make
> the queues very large. We've seen queues set to max. 200GB to be able to
> hold whole trading day of market data.
-------------- next part --------------
An HTML attachment was scrubbed...
More information about the zeromq-dev