[zeromq-dev] publish/subscribe and the "fast producer" problem

[Greg Young]

"You have to drop the generated events at some point if none of the
consumers can keep up, just need to figure out what is the best event
to drop."

Unless we assume a near-infinite (larger than what we care about)
buffer such as how kafka, eventstore, or chronicle work. This does
require a switch from push to pull model subscriptions in most cases.
As example "give me the next 10 messages after #2929292".

Cheers,

Greg

On Thu, Mar 9, 2017 at 3:35 PM, Max Kozlovsky <max at portworx.com> wrote:
> Hi,
>
> I think you are saying
>
> 1) you don't have to guarantee the delivery
> 2) You don't have control over the speed the events are generated
> 3) In event of some consumers not being able to keep up you want a best
> effort to deliver events to maximum consumers
>
> You have to drop the generated events at some point if none of the consumers
> can keep up, just need to figure out what is the best event to drop.
>
> It seems a pull model might be better in this case. Maintain a queue of
> generated events in the producer of arbitrary finite size with a sequence
> number. Have consumers pull the events from producer starting from the last
> sequence number they seen. If the queue becomes full, drop the oldest event.
> This gives the best chance for maximum number of consumers to process the
> events at their own speed.
>
> Max
>
> On Thu, Mar 9, 2017 at 5:56 AM, Francesco <francesco.montorsi at gmail.com>
> wrote:
>>
>> Hi Max,
>>
>> 2017-03-07 20:51 GMT+01:00 Max Kozlovsky <max at portworx.com>:
>> > I wonder what is the practical application of this model? If for
>> > correctness
>> > of the application it is enough to deliver every message to at least one
>> > of
>> > the subscribers, why not always sent just one copy of a message to just
>> > one
>> > of the subscribers?
>> No the correctness model is the following: assume that you have 10
>> subscribers. Suddenly 1 of them looses connectivity / crashes /
>> whatever. The other 9 are ok.
>> What happens is that:
>> a) If I use the NODROP=1, my publisher blocks and stops processing its
>> own data, eventually leading to a drop of its own inputs. The whole
>> system is now blocked (publisher dropping its input events [not coming
>> from network], 9 subscribers all waiting for the publisher to
>> publish).
>> b) If I use a currently non-existing ZMQ_XPUB_ATLEASTONE=1 option, my
>> publisher keeps publishing, 9 subscribers keep working, just 1 is
>> stopped / out of service. Outcome in my case: 90% of my application is
>> still working, I have 10% of data that is not processed and lost.
>>
>> Which scenario is best? IMO it's b) :)
>>
>>
>> >Sends to the rest of the subscribers are overhead since
>> > they are not required for correctness?
>> They are required, but in case something goes bad I prefer the failure
>> of a part rather than the failure of everything.
>>
>> Francesco
>> _______________________________________________
>> zeromq-dev mailing list
>> zeromq-dev at lists.zeromq.org
>> https://lists.zeromq.org/mailman/listinfo/zeromq-dev
>
>
>
> _______________________________________________
> zeromq-dev mailing list
> zeromq-dev at lists.zeromq.org
> https://lists.zeromq.org/mailman/listinfo/zeromq-dev



-- 
Studying for the Turing test