[zeromq-dev] publish/subscribe and the "fast producer" problem

Francesco francesco.montorsi at gmail.com
Thu Mar 9 19:48:22 CET 2017


Hi Max,
thanks for your reply. Just a question:

2017-03-09 16:35 GMT+01:00 Max Kozlovsky <max at portworx.com>:
> It seems a pull model might be better in this case.

from my understanding, mostly based on reading zeroMQ - the guide
(http://zguide.zeromq.org/page:all), the PUSH/PULL model is for
load-balancing, so that if I change my SUBscribers to be PULLers, and
my PUBlisher to be a PUSHer, the messages sent by the pusher will be
load-balanced over the pullers... that is each puller will get a
different message... I need (possibly) that all pullers get the same
message...

did I get it right?

Thanks,
Francesco


>Maintain a queue of
> generated events in the producer of arbitrary finite size with a sequence
> number. Have consumers pull the events from producer starting from the last
> sequence number they seen. If the queue becomes full, drop the oldest event.
> This gives the best chance for maximum number of consumers to process the
> events at their own speed.
>
> Max
>
> On Thu, Mar 9, 2017 at 5:56 AM, Francesco <francesco.montorsi at gmail.com>
> wrote:
>>
>> Hi Max,
>>
>> 2017-03-07 20:51 GMT+01:00 Max Kozlovsky <max at portworx.com>:
>> > I wonder what is the practical application of this model? If for
>> > correctness
>> > of the application it is enough to deliver every message to at least one
>> > of
>> > the subscribers, why not always sent just one copy of a message to just
>> > one
>> > of the subscribers?
>> No the correctness model is the following: assume that you have 10
>> subscribers. Suddenly 1 of them looses connectivity / crashes /
>> whatever. The other 9 are ok.
>> What happens is that:
>> a) If I use the NODROP=1, my publisher blocks and stops processing its
>> own data, eventually leading to a drop of its own inputs. The whole
>> system is now blocked (publisher dropping its input events [not coming
>> from network], 9 subscribers all waiting for the publisher to
>> publish).
>> b) If I use a currently non-existing ZMQ_XPUB_ATLEASTONE=1 option, my
>> publisher keeps publishing, 9 subscribers keep working, just 1 is
>> stopped / out of service. Outcome in my case: 90% of my application is
>> still working, I have 10% of data that is not processed and lost.
>>
>> Which scenario is best? IMO it's b) :)
>>
>>
>> >Sends to the rest of the subscribers are overhead since
>> > they are not required for correctness?
>> They are required, but in case something goes bad I prefer the failure
>> of a part rather than the failure of everything.
>>
>> Francesco
>> _______________________________________________
>> zeromq-dev mailing list
>> zeromq-dev at lists.zeromq.org
>> https://lists.zeromq.org/mailman/listinfo/zeromq-dev
>
>
>
> _______________________________________________
> zeromq-dev mailing list
> zeromq-dev at lists.zeromq.org
> https://lists.zeromq.org/mailman/listinfo/zeromq-dev



More information about the zeromq-dev mailing list