[zeromq-dev] PUB/SUB cached queue list

[Pieter Hintjens]

The pub-sub pattern is meant for high-volume unbreakable data
distribution. In this use case there is literally no recovery from a
slow subscriber except to drop messages. Any other strategy will
create latency spikes, or memory overflows on the publisher.

You can lose messages at the subscriber side too, if the application
thread is too slow in pulling them off the sub socket.

If you need guaranteed delivery you cannot use PUB-SUB. You can use
ROUTER-DEALER and add your own acknowledgements.

There's a lot of discussion of the different strategies and tradeoffs
in the Guide.

-Pieter

On Thu, Sep 26, 2013 at 6:15 PM, Gerry Steele <gerry.steele at gmail.com> wrote:
> In the event of HWM most middlewares provide an async callback to
> notify about the event... does ZMQ have anything like that? I see the
> same "issue"/behaviour as the parent and have a very large value set
> for my HWM on pub and sub but i see no memory growth that i might
> associate with a backlog.
>
> On 26 September 2013 17:01, Artem Vysochyn <artem.vysochyn at gmail.com> wrote:
>> hi Renato,
>>
>> ZMQ api manual is very clear about PUB/SUB:
>>>>
>> A socket of type ZMQ_PUB is used by a publisher to distribute data.
>> Messages sent are distributed in a fan out fashion to all connected
>> peers.
>> When a ZMQ_PUB socket enters the mute state due to having reached the
>> high water mark for a subscriber, then any messages that would be sent
>> to the subscriber in question shall instead be dropped until the mute
>> state ends.
>> <<
>> I'm afraid it's not correct to say "we lost messages", because it
>> doesn't reflect the reality. And reality in your system may be as
>> following:
>> 1.1 SUB socket(s)  _are not_ connected to PUB socket.
>> 1.2 SUB socket(s)  _were connected_ but then disconnected from PUB socket.
>> 1.3 SUB socket(s) reached HWM.
>>
>>
>> You had said that : >> When we loose messages we have no means to
>> verify this event. <<
>> Actually, you have means -- via poller.  Create a  poller, register
>> your PUB socket  for  OUT-going events, then poll(), then if
>> poller.pollout()  == true   then send a message.
>> At least this works for me. I'm on java, using jzmq JNI binding.   Works nicely.
>>
>>
>> 2013/9/26 gonzalo diethelm <gdiethelm at dcv.cl>
>>>
>>> Hi Renato,
>>>
>>>
>>>
>>> Your question falls under the broad concept of “reliability”. It seems pub/sub might not be a good fit for our requirements. Whether that is the case or not, if you want to get close to “no message loss” (or to detect when you have lost a message), you will need to add some mechanisms to your implementation (unique ids, increasing timestamps, idempotency, etc.).
>>>
>>>
>>>
>>> Vast, vast ground for discussion… Good luck!
>>>
>>>
>>>
>>> --
>>>
>>> Gonzalo Diethelm
>>>
>>> DCV Chile
>>>
>>>
>>>
>>> From: zeromq-dev-bounces at lists.zeromq.org [mailto:zeromq-dev-bounces at lists.zeromq.org] On Behalf Of Renato Samperio
>>> Sent: Thursday, September 26, 2013 11:42 AM
>>> To: zeromq-dev at lists.zeromq.org
>>> Subject: [zeromq-dev] PUB/SUB cached queue list
>>>
>>>
>>>
>>> Dear all,
>>>
>>> I have been trying to follow the list messages over the last months except for last two weeks. My apologies if I have skipped this issue in any message.
>>>
>>> We have written our solution in ZMQ and we have tested it with a PUB/SUB pattern implementation.
>>>
>>> The big trouble we have is that depending on the system hardware architecture (speed and memory resources) we can loose messages. When we loose messages we have no means to verify this event.
>>>
>>> By the way, for the requirements of our system we must not loose messages or have delays in their delivery.
>>>
>>> ZMQ does not have any support for checking the queue status (at both PUB and SUBs sides), or whenever we publish a message if this message is lost (because ZMQ silently throws away it) when the queue is full.
>>>
>>> Is there a way to?
>>> - know status of the queues, at any time;
>>> - get back the info that a message is/is-not queued whenever a publish is done.
>>>
>>>
>>> We would also like to set a ticket for such issues. Could you point us where is the right place to do it?
>>>
>>> Thanks for your help...
>>>
>>>
>>> Regards,
>>>
>>> Renato.
>>>
>>> ________________________________
>>>
>>> Declaración de confidencialidad: Este Mensaje esta destinado para el uso de la o las personas o entidades a quien ha sido dirigido y puede contener información reservada y confidencial que no puede ser divulgada, difundida, ni aprovechada en forma alguna. El uso no autorizado de la información contenida en este correo podrá ser sancionado de conformidad con la ley chilena. Si usted ha recibido este correo electrónico por error, le pedimos eliminarlo junto con los archivos adjuntos y avisar inmediatamente al remitente, respondiendo este mensaje. Disclosure: This Message is to be used by the individual, individuals or entities that it is addressed to and may include private and confidential information that may not be disclosed, made public nor used in any way at all. Unauthorized use of the information in this electronic mail message may be subject to the penalties set forth by Chilean law. If you have received this electronic mail message in error, we ask you to destroy the message and its attached file(s) and to immediately notify the sender by answering this message.
>>>
>>> _______________________________________________
>>> zeromq-dev mailing list
>>> zeromq-dev at lists.zeromq.org
>>> http://lists.zeromq.org/mailman/listinfo/zeromq-dev
>>>
>> _______________________________________________
>> zeromq-dev mailing list
>> zeromq-dev at lists.zeromq.org
>> http://lists.zeromq.org/mailman/listinfo/zeromq-dev
>
>
>
> --
> Gerry Steele
> _______________________________________________
> zeromq-dev mailing list
> zeromq-dev at lists.zeromq.org
> http://lists.zeromq.org/mailman/listinfo/zeromq-dev