[zeromq-dev] How zeromq handles slow consumers with PUB/SUB mode

[joel vennin]

Thank you for these useful information !

On Tue, Mar 23, 2010 at 4:33 PM, Martin Sustrik <sustrik at 250bpm.com> wrote:

> Hi Joel,
>
>
>  I'm planning to write a prototype using zeromq. I see in the documentation
>> that zeromq provides a publisher/subscriber mode. I want to use this mode in
>> my application, but i'm asking how zeromq handles
>> slow consumer.
>>
>> Is there some documentation about this point ?
>>
>
> The documentation is lacking in this area :(
>
> However, the principle is pretty simple: You can think of the network as
> series of tubes. Each tube has some max capacity. For example, when
> communicating using 2 0MQ sockets there's senders 0MQ queue, kernel SNDBUF,
> queue in the NIC, queue in the switch, queue in the receiver's NIC, kernel
> RCVBUF, receiver's 0MQ queue...
>
> When consumer is slow these queues start growing. The process starts at the
> place of the bottleneck then propagates up back to the sender. Once all the
> queues are full, meaning that sender cannot send another message without
> exceeding the limits, some measure must be taken.
>
> The exact measure depends on the messaging pattern. For example for REQ/REP
> send function blocks in such a situation. For PUB/SUB the message is
> accepted and dropped.
>
> While the latter looks scary, it makes perfect sense because blocking would
> cause whole system to stop. Even innocent subscribers would not get
> messages.
>
> If you want to reasonably guarantee the delivery, the only way is to make
> the queues very large. We've seen queues set to max. 200GB to be able to
> hold whole trading day of market data.
>
> Martin
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.zeromq.org/pipermail/zeromq-dev/attachments/20100323/c40ee075/attachment.htm>