[zeromq-dev] publish/subscribe and the "fast producer" problem

[Max Kozlovsky]

Hi,

I think you are saying

1) you don't have to guarantee the delivery
2) You don't have control over the speed the events are generated
3) In event of some consumers not being able to keep up you want a best
effort to deliver events to maximum consumers

You have to drop the generated events at some point if none of the
consumers can keep up, just need to figure out what is the best event to
drop.

It seems a pull model might be better in this case. Maintain a queue of
generated events in the producer of arbitrary finite size with a sequence
number. Have consumers pull the events from producer starting from the last
sequence number they seen. If the queue becomes full, drop the oldest
event. This gives the best chance for maximum number of consumers to
process the events at their own speed.

Max

On Thu, Mar 9, 2017 at 5:56 AM, Francesco <francesco.montorsi at gmail.com>
wrote:

> Hi Max,
>
> 2017-03-07 20:51 GMT+01:00 Max Kozlovsky <max at portworx.com>:
> > I wonder what is the practical application of this model? If for
> correctness
> > of the application it is enough to deliver every message to at least one
> of
> > the subscribers, why not always sent just one copy of a message to just
> one
> > of the subscribers?
> No the correctness model is the following: assume that you have 10
> subscribers. Suddenly 1 of them looses connectivity / crashes /
> whatever. The other 9 are ok.
> What happens is that:
> a) If I use the NODROP=1, my publisher blocks and stops processing its
> own data, eventually leading to a drop of its own inputs. The whole
> system is now blocked (publisher dropping its input events [not coming
> from network], 9 subscribers all waiting for the publisher to
> publish).
> b) If I use a currently non-existing ZMQ_XPUB_ATLEASTONE=1 option, my
> publisher keeps publishing, 9 subscribers keep working, just 1 is
> stopped / out of service. Outcome in my case: 90% of my application is
> still working, I have 10% of data that is not processed and lost.
>
> Which scenario is best? IMO it's b) :)
>
>
> >Sends to the rest of the subscribers are overhead since
> > they are not required for correctness?
> They are required, but in case something goes bad I prefer the failure
> of a part rather than the failure of everything.
>
> Francesco
> _______________________________________________
> zeromq-dev mailing list
> zeromq-dev at lists.zeromq.org
> https://lists.zeromq.org/mailman/listinfo/zeromq-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.zeromq.org/pipermail/zeromq-dev/attachments/20170309/f6bf99de/attachment.htm>