[zeromq-dev] maintaining order of messages while having no single point of failiure

Kevin Sapper kevinsapper88 at gmail.com
Sat Jun 6 15:47:04 CEST 2020


Hi Nir,

we created the Dafka protocol which fulfills many of your requirements.
Have a look at the reference implementation and the rfc.

https://github.com/zeromq/dafka


https://rfc.zeromq.org/spec/46/

Cheers
Kevin

Nir Sharony <nsharony at gmail.com> schrieb am Sa., 6. Juni 2020, 15:40:

> Hi, guys.
>
> I wanted to share with you some design requirements and see if you could
> point me in the right direction.
>
> I am developing a system that consists of multiple microservices that need
> to communicate among themselves. Microservices belong in groups. In order
> to not have a single point of failure and to handle a large volume of
> messages, each group will contain at least two (probably more)
> microservices.
>
> When a microservice publishes a message, it designated to a specific group
> and should be processed by just one microservice within that group. Note
> that if the processing fails (microservice dies in the middle), the same
> message should be picked up some other microservice in the same group.
>
> An additional requirement in my system is that the order of the messages
> that are being processing by a group, must be maintained because the
> internal state relies on it. While this is true for all the message
> consumers in the system, in practicality, the chance of messages getting
> processed out of order highly depends on the pace that they are being
> produced and consumed.
>
> Some messages are produced at a low rate (one message every 5-10 seconds).
> As the processing time for each message is just a few milliseconds, the
> chance that consumer A will finish processing message X *AFTER* consumer
> B finished processing message X+1 is slim.
> In such cases, we can probably lift that "maintain message order"
> requirement and we will still be ok.
>
> However, some producers produce messages at a much higher pace (sometimes
> more than 30 times a second). This means that the chance the consumers are
> processing those jobs out of order increases.
> In such cases, the "maintain message order" cannot be ignored.
>
> Does ZeroMQ address these issues at all?
>
> Thanks in advance.
> _______________________________________________
> zeromq-dev mailing list
> zeromq-dev at lists.zeromq.org
> https://lists.zeromq.org/mailman/listinfo/zeromq-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.zeromq.org/pipermail/zeromq-dev/attachments/20200606/87393e9f/attachment.htm>


More information about the zeromq-dev mailing list