[zeromq-dev] publish/subscribe and the "fast producer" problem
Max Kozlovsky
max at portworx.com
Thu Mar 9 16:47:21 CET 2017
Exactly.
On Thu, Mar 9, 2017 at 7:40 AM, Greg Young <gregoryyoung1 at gmail.com> wrote:
> "You have to drop the generated events at some point if none of the
> consumers can keep up, just need to figure out what is the best event
> to drop."
>
> Unless we assume a near-infinite (larger than what we care about)
> buffer such as how kafka, eventstore, or chronicle work. This does
> require a switch from push to pull model subscriptions in most cases.
> As example "give me the next 10 messages after #2929292".
>
> Cheers,
>
> Greg
>
> On Thu, Mar 9, 2017 at 3:35 PM, Max Kozlovsky <max at portworx.com> wrote:
> > Hi,
> >
> > I think you are saying
> >
> > 1) you don't have to guarantee the delivery
> > 2) You don't have control over the speed the events are generated
> > 3) In event of some consumers not being able to keep up you want a best
> > effort to deliver events to maximum consumers
> >
> > You have to drop the generated events at some point if none of the
> consumers
> > can keep up, just need to figure out what is the best event to drop.
> >
> > It seems a pull model might be better in this case. Maintain a queue of
> > generated events in the producer of arbitrary finite size with a sequence
> > number. Have consumers pull the events from producer starting from the
> last
> > sequence number they seen. If the queue becomes full, drop the oldest
> event.
> > This gives the best chance for maximum number of consumers to process the
> > events at their own speed.
> >
> > Max
> >
> > On Thu, Mar 9, 2017 at 5:56 AM, Francesco <francesco.montorsi at gmail.com>
> > wrote:
> >>
> >> Hi Max,
> >>
> >> 2017-03-07 20:51 GMT+01:00 Max Kozlovsky <max at portworx.com>:
> >> > I wonder what is the practical application of this model? If for
> >> > correctness
> >> > of the application it is enough to deliver every message to at least
> one
> >> > of
> >> > the subscribers, why not always sent just one copy of a message to
> just
> >> > one
> >> > of the subscribers?
> >> No the correctness model is the following: assume that you have 10
> >> subscribers. Suddenly 1 of them looses connectivity / crashes /
> >> whatever. The other 9 are ok.
> >> What happens is that:
> >> a) If I use the NODROP=1, my publisher blocks and stops processing its
> >> own data, eventually leading to a drop of its own inputs. The whole
> >> system is now blocked (publisher dropping its input events [not coming
> >> from network], 9 subscribers all waiting for the publisher to
> >> publish).
> >> b) If I use a currently non-existing ZMQ_XPUB_ATLEASTONE=1 option, my
> >> publisher keeps publishing, 9 subscribers keep working, just 1 is
> >> stopped / out of service. Outcome in my case: 90% of my application is
> >> still working, I have 10% of data that is not processed and lost.
> >>
> >> Which scenario is best? IMO it's b) :)
> >>
> >>
> >> >Sends to the rest of the subscribers are overhead since
> >> > they are not required for correctness?
> >> They are required, but in case something goes bad I prefer the failure
> >> of a part rather than the failure of everything.
> >>
> >> Francesco
> >> _______________________________________________
> >> zeromq-dev mailing list
> >> zeromq-dev at lists.zeromq.org
> >> https://lists.zeromq.org/mailman/listinfo/zeromq-dev
> >
> >
> >
> > _______________________________________________
> > zeromq-dev mailing list
> > zeromq-dev at lists.zeromq.org
> > https://lists.zeromq.org/mailman/listinfo/zeromq-dev
>
>
>
> --
> Studying for the Turing test
> _______________________________________________
> zeromq-dev mailing list
> zeromq-dev at lists.zeromq.org
> https://lists.zeromq.org/mailman/listinfo/zeromq-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.zeromq.org/pipermail/zeromq-dev/attachments/20170309/24eaa415/attachment.htm>
More information about the zeromq-dev
mailing list