[zeromq-dev] maintaining order of messages while having no single point of failiure

Nir Sharony nsharony at gmail.com
Sat Jun 6 15:37:40 CEST 2020


Hi, guys.

I wanted to share with you some design requirements and see if you could
point me in the right direction.

I am developing a system that consists of multiple microservices that need
to communicate among themselves. Microservices belong in groups. In order
to not have a single point of failure and to handle a large volume of
messages, each group will contain at least two (probably more)
microservices.

When a microservice publishes a message, it designated to a specific group
and should be processed by just one microservice within that group. Note
that if the processing fails (microservice dies in the middle), the same
message should be picked up some other microservice in the same group.

An additional requirement in my system is that the order of the messages
that are being processing by a group, must be maintained because the
internal state relies on it. While this is true for all the message
consumers in the system, in practicality, the chance of messages getting
processed out of order highly depends on the pace that they are being
produced and consumed.

Some messages are produced at a low rate (one message every 5-10 seconds).
As the processing time for each message is just a few milliseconds, the
chance that consumer A will finish processing message X *AFTER* consumer B
finished processing message X+1 is slim.
In such cases, we can probably lift that "maintain message order"
requirement and we will still be ok.

However, some producers produce messages at a much higher pace (sometimes
more than 30 times a second). This means that the chance the consumers are
processing those jobs out of order increases.
In such cases, the "maintain message order" cannot be ignored.

Does ZeroMQ address these issues at all?

Thanks in advance.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.zeromq.org/pipermail/zeromq-dev/attachments/20200606/78c5c842/attachment.htm>


More information about the zeromq-dev mailing list