[zeromq-dev] Question about reliability in ZeroMQ
白木勇矢
white.tree at keio.jp
Wed Oct 19 18:57:02 CEST 2016
I see. Thank you Joshua.
Yuya
On Tue, Oct 18, 2016 at 3:52 PM, Joshua Foster <jhawk28 at gmail.com> wrote:
> Yes, the old data still in the queue would be sent once the dealer starts
> receiving messages again.
>
> Joshua
>
>
>
> On 10/18/2016 6:08 PM, 白木 勇矢 wrote:
>
>> Thank you for your response Joshua. So I cannot avoid queue drop when
>> sending packet from Pub to Sub or from Dealer to Router unless I
>> create some logic to solve this issue.
>>
>> According to what you explained, when I call zmq_send continuously on
>> Router to send messages to specific Dealer, if Dealer is in mute
>> state, because the Router socket send is blocked and cannot dequeue a
>> message, it's going to reach high water mark in Router. After it
>> reached high water mark in Router, rest of the message I tried to
>> enqueue using zmq_send in Router is dropped. But do the previous data
>> enqueued already in ZMQ queue of Router would be able to reach Dealer
>> for sure if Dealer move out from mute state?
>>
>> Thanks,
>> Yuya
>>
>> On Tue, Oct 18, 2016 at 10:48 AM, Joshua Foster <jhawk28 at gmail.com>
>> wrote:
>>
>>> Lets see if I can explain it. Each socket has its own queue/buffer. This
>>> is
>>> what is used to track how many messages. PUB/SUB uses a drop approach
>>> when
>>> its queue gets full. If a subscriber is slow (has a full queue), the
>>> publisher continues publishing and does not block. The slow subscriber
>>> would
>>> then miss all messages until it was able to clear some of its queue.
>>>
>>> I don't know much about PGM, but I believe that it just provides more
>>> reliability at the packet level so the behavior of PUB/SUB would be the
>>> same.
>>>
>>> PUSH/PULL block when their queues get full. This puts back-pressure on
>>> the
>>> producers. Router and Dealers are sockets with a pair of Push/Pull
>>> sockets
>>> so they behave the same. So if a Dealer is blocked, the router socket
>>> would
>>> block on send until the dealer can accept.
>>>
>>> The correct answer to avoiding queue drops is that you don't. The correct
>>> approach is to create you solution that allows messages to drop. There
>>> are
>>> MANY reasons for messages to drop and it is really just a tradeoff as to
>>> what can drop and what performance is required. One simple way to handle
>>> dropped messages is to use idempotent messages. The basic idea is that
>>> you
>>> create messages that contain the most recent state of whatever you need.
>>> That way if all previous messages get dropped, you still have all the
>>> data
>>> needed. In contrast, transactional messages are not able to be dropped.
>>>
>>> Idempotent Example:
>>> {1}
>>> {5}
>>> {10}
>>>
>>> Transactional Example:
>>> {+1}
>>> {+4}
>>> {+5}
>>>
>>> Joshua
>>>
>>>
>>> On 10/18/2016 1:21 PM, 白木勇矢 wrote:
>>>
>>>> Hello everyone,
>>>>
>>>> I'm trying to use ZeroMQ Pub/Sub and Router/Dealer for my school
>>>> project. I have a question about reliability of these sockets.
>>>> According to the API Page(http://api.zeromq.org/), when queue reached
>>>> a high water mark and move to mute state, Dealer is going to block,
>>>> Router is going to drop, Sub I'm not sure, and Pub is going to drop
>>>> the message. But I cannot clearly see the behavior of each socket.
>>>>
>>>> When Dealer send message to Router via TCP and Router is in mute
>>>> state, Dealer would be able to know this and resend message again
>>>> after some interval? If Router sends the message and Dealer is in mute
>>>> state and blocked, how Router is going to act?
>>>>
>>>> Also when Pub sends the message to Sub via TCP and if Sub is in mute
>>>> state, how Pub is going to behave? If Pub wants to know whether the
>>>> message is correctly enqueued in Sub, PGM is going to solve this?
>>>>
>>>> If someone would be able to answer these questions and tell me how to
>>>> avoid queue drops, I appreciate it so much.
>>>>
>>>> Thanks,
>>>> Yuya
>>>> _______________________________________________
>>>> zeromq-dev mailing list
>>>> zeromq-dev at lists.zeromq.org
>>>> http://lists.zeromq.org/mailman/listinfo/zeromq-dev
>>>>
>>>
>>> _______________________________________________
>>> zeromq-dev mailing list
>>> zeromq-dev at lists.zeromq.org
>>> http://lists.zeromq.org/mailman/listinfo/zeromq-dev
>>>
>> _______________________________________________
>> zeromq-dev mailing list
>> zeromq-dev at lists.zeromq.org
>> http://lists.zeromq.org/mailman/listinfo/zeromq-dev
>>
>
> _______________________________________________
> zeromq-dev mailing list
> zeromq-dev at lists.zeromq.org
> http://lists.zeromq.org/mailman/listinfo/zeromq-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.zeromq.org/pipermail/zeromq-dev/attachments/20161019/c7e89870/attachment.htm>
More information about the zeromq-dev
mailing list