[zeromq-dev] Question about reliability in ZeroMQ

[Joshua Foster]

Yes, the old data still in the queue would be sent once the dealer 
starts receiving messages again.

Joshua


On 10/18/2016 6:08 PM, 白木 勇矢 wrote:
> Thank you for your response Joshua. So I cannot avoid queue drop when
> sending packet from Pub to Sub or from Dealer to Router unless I
> create some logic to solve this issue.
>
> According to what you explained, when I call zmq_send continuously on
> Router to send messages to specific Dealer, if Dealer is in mute
> state, because the Router socket send is blocked and cannot dequeue a
> message, it's going to reach high water mark in Router. After it
> reached high water mark in Router, rest of the message I tried to
> enqueue using zmq_send in Router is dropped. But do the previous data
> enqueued already in ZMQ queue of Router would be able to reach Dealer
> for sure if Dealer move out from mute state?
>
> Thanks,
> Yuya
>
> On Tue, Oct 18, 2016 at 10:48 AM, Joshua Foster <jhawk28 at gmail.com> wrote:
>> Lets see if I can explain it. Each socket has its own queue/buffer. This is
>> what is used to track how many messages. PUB/SUB uses a drop approach when
>> its queue gets full. If a subscriber is slow (has a full queue), the
>> publisher continues publishing and does not block. The slow subscriber would
>> then miss all messages until it was able to clear some of its queue.
>>
>> I don't know much about PGM, but I believe that it just provides more
>> reliability at the packet level so the behavior of PUB/SUB would be the
>> same.
>>
>> PUSH/PULL block when their queues get full. This puts back-pressure on the
>> producers. Router and Dealers are sockets with a pair of Push/Pull sockets
>> so they behave the same. So if a Dealer is blocked, the router socket would
>> block on send until the dealer can accept.
>>
>> The correct answer to avoiding queue drops is that you don't. The correct
>> approach is to create you solution that allows messages to drop. There are
>> MANY reasons for messages to drop and it is really just a tradeoff as to
>> what can drop and what performance is required. One simple way to handle
>> dropped messages is to use idempotent messages. The basic idea is that you
>> create messages that contain the most recent state of whatever you need.
>> That way if all previous messages get dropped, you still have all the data
>> needed. In contrast, transactional messages are not able to be dropped.
>>
>> Idempotent Example:
>> {1}
>> {5}
>> {10}
>>
>> Transactional Example:
>> {+1}
>> {+4}
>> {+5}
>>
>> Joshua
>>
>>
>> On 10/18/2016 1:21 PM, 白木勇矢 wrote:
>>> Hello everyone,
>>>
>>> I'm trying to use ZeroMQ Pub/Sub and Router/Dealer for my school
>>> project. I have a question about reliability of these sockets.
>>> According to the API Page(http://api.zeromq.org/), when queue reached
>>> a high water mark and move to mute state, Dealer is going to block,
>>> Router is going to drop, Sub I'm not sure, and Pub is going to drop
>>> the message. But I cannot clearly see the behavior of each socket.
>>>
>>> When Dealer send message to Router via TCP and Router is in mute
>>> state, Dealer would be able to know this and resend message again
>>> after some interval? If Router sends the message and Dealer is in mute
>>> state and blocked, how Router is going to act?
>>>
>>> Also when Pub sends the message to Sub via TCP and if Sub is in mute
>>> state, how Pub is going to behave? If Pub wants to know whether the
>>> message is correctly enqueued in Sub, PGM is going to solve this?
>>>
>>> If someone would be able to answer these questions and tell me how to
>>> avoid queue drops, I appreciate it so much.
>>>
>>> Thanks,
>>> Yuya
>>> _______________________________________________
>>> zeromq-dev mailing list
>>> zeromq-dev at lists.zeromq.org
>>> http://lists.zeromq.org/mailman/listinfo/zeromq-dev
>>
>> _______________________________________________
>> zeromq-dev mailing list
>> zeromq-dev at lists.zeromq.org
>> http://lists.zeromq.org/mailman/listinfo/zeromq-dev
> _______________________________________________
> zeromq-dev mailing list
> zeromq-dev at lists.zeromq.org
> http://lists.zeromq.org/mailman/listinfo/zeromq-dev