[zeromq-dev] Question about reliability in ZeroMQ

[Joshua Foster]

Lets see if I can explain it. Each socket has its own queue/buffer. This 
is what is used to track how many messages. PUB/SUB uses a drop approach 
when its queue gets full. If a subscriber is slow (has a full queue), 
the publisher continues publishing and does not block. The slow 
subscriber would then miss all messages until it was able to clear some 
of its queue.

I don't know much about PGM, but I believe that it just provides more 
reliability at the packet level so the behavior of PUB/SUB would be the 
same.

PUSH/PULL block when their queues get full. This puts back-pressure on 
the producers. Router and Dealers are sockets with a pair of Push/Pull 
sockets so they behave the same. So if a Dealer is blocked, the router 
socket would block on send until the dealer can accept.

The correct answer to avoiding queue drops is that you don't. The 
correct approach is to create you solution that allows messages to drop. 
There are MANY reasons for messages to drop and it is really just a 
tradeoff as to what can drop and what performance is required. One 
simple way to handle dropped messages is to use idempotent messages. The 
basic idea is that you create messages that contain the most recent 
state of whatever you need. That way if all previous messages get 
dropped, you still have all the data needed. In contrast, transactional 
messages are not able to be dropped.

Idempotent Example:
{1}
{5}
{10}

Transactional Example:
{+1}
{+4}
{+5}

Joshua

On 10/18/2016 1:21 PM, 白木勇矢 wrote:
> Hello everyone,
>
> I'm trying to use ZeroMQ Pub/Sub and Router/Dealer for my school
> project. I have a question about reliability of these sockets.
> According to the API Page(http://api.zeromq.org/), when queue reached
> a high water mark and move to mute state, Dealer is going to block,
> Router is going to drop, Sub I'm not sure, and Pub is going to drop
> the message. But I cannot clearly see the behavior of each socket.
>
> When Dealer send message to Router via TCP and Router is in mute
> state, Dealer would be able to know this and resend message again
> after some interval? If Router sends the message and Dealer is in mute
> state and blocked, how Router is going to act?
>
> Also when Pub sends the message to Sub via TCP and if Sub is in mute
> state, how Pub is going to behave? If Pub wants to know whether the
> message is correctly enqueued in Sub, PGM is going to solve this?
>
> If someone would be able to answer these questions and tell me how to
> avoid queue drops, I appreciate it so much.
>
> Thanks,
> Yuya
> _______________________________________________
> zeromq-dev mailing list
> zeromq-dev at lists.zeromq.org
> http://lists.zeromq.org/mailman/listinfo/zeromq-dev