[zeromq-dev] distributed message bus on top of / built with zeromq?

[Sabri Skhiri]

Hi Holger,

Not sure it is the right place to discuss about RoQ since it is not
directly linked to ZMQ ( except that we use it for as basic components of
the Elastic Bus). So, I will answer your question but do not hesitate to
continue the discussion "off-list".

- Does one EQS queue cover multiple topics (or subjects)?
Yes it does, we use the underlying ZMQ pub/sub topic-based routing.

- Does one EQS exchange cover multiple topics/subjects?
Yes as it is the ZMQ pub sub which filters.

- Is a producer of 1 topic/subject connected to one exchange?
I do not really get the question, if you asked whether 2 producers of 2
different topics must be assigned to 2 different exchange, I would answer
"not especially", you can have different producers sending on different
topics connected on the same exchange.

- Are 2 listeners on the same topic/subject always connected to the same
exchange?
The listeners must be connected to all exchanges of 1 logical queue. This
is part of the things we want to make evolve.

- During relocation of a producer, I take it that listeners don't lose
messages (producer probably
stops sending messages during relocation)?
Currently we rely on the internal ZMQ internal buffers.

- Did you define any wire protocol for RoQ?
No, we only use the simple sendMessage Semantics and we rely on a "Pub->
Exchange-> Listeners" pattern.

- With regard to your elastic scaling algorithm:
What happens if one single producer overloads an exchange all by itself?
This is the limit case. We have been thinking about this for a while, the
simplest solution would be to locally (at the pub side)  divide the
publication stream into sub-publishers, each of them could then be
relocated according to the existing re-location logic.

- I understand RoQ uses TCP/IP for transport. Do you have any plans on
using UDP multicast? How
would you scale for a great number of listeners for one topic/subject?
This is exactly why we have proposed the evolution toward a better topic
management. (in the roadmap page).

- What kind of QoS or delivery guarantee are you targetting for your future
plans on persistence and
reliability, e.g. at-least-once, exactly-once,...?
We do not support any QoS currently. We are currently working on two
aspects: introducing simple HA (High Availability)  mechanism for
supporting the HA. The last bunch of pull requests are a first step in this
direction.  Secondly, we are looking at different HA strategy in the state
of the art in NoSQL storage, Stream Processing and Data Processing in order
to define the best strategy that could be applied in a  messaging context.

- Is my understanding correct that the RoQ management components are single
points of failure of
the RoQ infrastructure and needed to be clustered / hot standby?
Indeed, that is also part of the last bunch of pull requests.

- Citing your future plans:
"[...] UC3 - Reconnected subscriber: if a subscriber disconnect from the
topic it must be able to either (1)
receive all the events from the beginning of the topic, (2) receive all the
events from the last connection or
(3) receive no historical events and just starts listening new ones. For
the case (2) we take as assumption
that the client must keep the state of its last connection (last message
sequence ID for instance).[...]"
Do you plan on implementing this in the subscriber client API layer or will
this be the responsibility of the
subscriber application?
At the API layer.

- Is my understanding correct that RoQ is currently Java-only? Are the API
calls and interactions
formally specified for development of non-Java clients?
This is JAVA only, but for a lot of components, we use ZMQ and BSON (for
the exchanges but also for the management), so for them, you can use any
ZMQ binding with the right BSON decoder.

I hope it answers your questions. Do not hesitate to continue the
discussion off-list.

Best regards,

Sabri.

2014-09-22 14:38 GMT+02:00 Holger Joukl <Holger.Joukl at lbbw.de>:

> Hi Sabri,
>
> zeromq-dev-bounces at lists.zeromq.org schrieb am 18.09.2014 19:07:02:
>
> > Von: Sabri Skhiri <sabri.skhiri at gmail.com>
> > We detected the necessity
> > to create a lightweight and elastic bus on top of ZeroMQ in the end of
> > 2010. At that time we started an internal research project and we
> > published our first conclusion in 2011 at the IEEE CloudCom conference
> > (http://euranova.eu/download?type=publications&title=EQS%3A+AN
> > +ELASTIC+AND+SCALABLE+MESSAGE+QUEUE+FOR+THE+CLOUD).
> > The paper explains how we designed an elastic message bus on top of
> > ZMQ.
> > After the publication, the compmunity asked for shaing the code and we
> > open Sourced RoQ (http://roq-messaging.org/). Today, this is still a
> > proof of concept and not a product yet, however we still use it within
> > EURA NOVA for going further in the understanding of all the technical
> > aspects of a distributed message bus (HA, reliability, scalability,
> > etc.).
>
> Thanks for sharing this information! This certainly looks interesting.
>
> I've now read your EQS paper
> (http://roq-messaging.org/docs/EQS.pdf) and looked at your future plans
> (https://github.com/roq-messaging/RoQ/wiki/Roadmap-%26-Milestones)
>
> ...and I do have some questions :-):
>
> - Does one EQS queue cover multiple topics (or subjects)?
>
> - Does one EQS exchange cover multiple topics/subjects?
>
> - Is a producer of 1 topic/subject connected to one exchange?
>
> - Are 2 listeners on the same topic/subject always connected to the same
> exchange?
>
> - During relocation of a producer, I take it that listeners don't lose
> messages (producer probably
> stops sending messages during relocation)?
>
> - Did you define any wire protocol for RoQ?
>
> - With regard to your elastic scaling algorithm:
> What happens if one single producer overloads an exchange all by itself?
>
> - I understand RoQ uses TCP/IP for transport. Do you have any plans on
> using UDP multicast? How
> would you scale for a great number of listeners for one topic/subject?
>
> - What kind of QoS or delivery guarantee are you targetting for your future
> plans on persistence and
> reliability, e.g. at-least-once, exactly-once,...?
>
> - Is my understanding correct that the RoQ management components are single
> points of failure of
> the RoQ infrastructure and needed to be clustered / hot standby?
>
> - Citing your future plans:
> "[...] UC3 - Reconnected subscriber: if a subscriber disconnect from the
> topic it must be able to either (1)
> receive all the events from the beginning of the topic, (2) receive all the
> events from the last connection or
> (3) receive no historical events and just starts listening new ones. For
> the case (2) we take as assumption
> that the client must keep the state of its last connection (last message
> sequence ID for instance).[...]"
> Do you plan on implementing this in the subscriber client API layer or will
> this be the responsibility of the
> subscriber application?
>
> - Is my understanding correct that RoQ is currently Java-only? Are the API
> calls and interactions
> formally specified for development of non-Java clients?
>
> Best regards
> Holger
>
> Landesbank Baden-Wuerttemberg
> Anstalt des oeffentlichen Rechts
> Hauptsitze: Stuttgart, Karlsruhe, Mannheim, Mainz
> HRA 12704
> Amtsgericht Stuttgart
>
> _______________________________________________
> zeromq-dev mailing list
> zeromq-dev at lists.zeromq.org
> http://lists.zeromq.org/mailman/listinfo/zeromq-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.zeromq.org/pipermail/zeromq-dev/attachments/20140923/4eaea342/attachment.htm>