[zeromq-dev] distributed message bus on top of / built with zeromq?

[Holger Joukl]

Dear all,

this is my first post to this list so please bear with my rather lengthy
thoughts.

I'm working in a financial markets environment and am currently looking
into
alternatives for near-realtime trade data message delivery (contracts or
deals, not market data ticks).

Traditionally we are using Tibco TIB/Rendezvous (and I've extensive
experience with it)
which has its merits of being mature and rock-solid, but it is truely
expensive.
Much the same holds true for IBM's WebsphereMQ which is also used in our
house already.

We need guaranteed or certified delivery for our data, ideally with a
quality of service of
exactly-once-delivery, At-least-once may also be possible if duplicate
delivery is a rare
cornercase. To the best of my knowledge even TIB/Rv doesn't provide
exactly-once-delivery though
the docs don't really explicitly state this, at least in its standard CM
(certified messaging) notion. But I've never
personally encountered duplicated RV message delivery in a decade.

Now, the obvious candidates for replacing the big beasts would be one of
the AMQP bunch but
of course all of those require a central broker/server. As I really like
TIB/Rv's distributed nature
I'm now evaluating the use of  zeromq (as in: toying around with the idea).

Alternatives that I can see:
1. Stay with TIB/Rv.
+ distributed architecture
+ proven battle-tested solution
+ reliable (in a sense of a company not about to vanish soon) commercial
support by its creators
- immense license and support fees.

2. Use one of the AMQP products (which?):
+ open source
+ open standard, so theoretically interchangeable implementations (see
below for standards issues)
+ commercial products available (e.g. Red Hat MRG with QPid under the hood,
or VMWare Pivotal RabbitMQ)
+ commercial support available
- central broker/server as a single point of failure (needs its own
HA/clustering/failover, dedicated monitoring,
dedicated administration)
- standardization process seems a mess and might actually have rendered the
original AMQP initiative
goals useless; though it looks like 1.0 now has rather widespread adoption
I somehow have a lurking
uncomfortable feeling

3. Roll your own solution, possibly based on zeromq
+ open source
+ open standard
+ full control over the architecture, can be fully distributed
+ flexibility, a lot of usage patterns
+ commercial support available (?)
- development effort, maybe massive (?)

Some additional alternatives might be: Moving this onto WebsphereMQ (also
expensive), using
Apache Kafka, maybe the Spread library (seems to have hard message size
limits).

I'm thinking about the possibilities of creating some kind of lightweight
distributed pub-sub message bus with
zeromq using multicast. Actually much like TIB/Rv conceptually so I wonder
a bit why nobody seems to have
done this yet with zeromq ;-). At least I'm not aware of it so I hope
that's not a totally inappropriate way
to use zeromq...?

Here goes:

1. (E)PGM or NORM for multicast? zeromq-NORM is not yet in the latest
release version of zeromq.

2. I understand it is not possible to use several publishers on one host
using the same multicast group
and port with (E)PGM due to NACKS only getting back to one publisher.
(referring e.g. to this thread:
http://grokbase.com/t/zeromq/zeromq-dev/127t25hcex/zmq-mcast-loop-with-pgm

But I'm confused by these posts in this thread:
http://grokbase.com/t/zeromq/zeromq-dev/127t25hcex/zmq-mcast-loop-with-pgm#2012080893pabdv3gkhg18wrqtbnp81gz8
http://grokbase.com/t/zeromq/zeromq-dev/127t25hcex/zmq-mcast-loop-with-pgm#20120809x9w6d4tw53aft99tgb1sf7mwy4

Is it impossible to have one publisher per host using the same multicast
group + port? Why?

Because if it actually is possible than it shouldn't be too hard to create
s.th. much like TIB/Rv's daemon approach, i.e. a bus
comprised of dedicated daemons on every host where the actually messaging
clients connect to. The daemons
handle all of the actual multicast communication.

Does NORM also have this restriction/caveat?

3. Guaranteed or certified delivery (QoS) would have to be layered as an
applicaton protocol on top of this. Again,
this could borrow from TIB/Rv concepts, i.e. publishers storing sent
messages persistently until they have been acknowledged
by all known/registered receivers.
Has anybody done s.th. like this already? I haven't been able to find
something like it, apart from some
general discussion on "reliability".

4. I understand zeromq does not impose limits on the message size. Is this
correct?

I'd be grateful for any insight.

Best regards
Holger

Landesbank Baden-Wuerttemberg
Anstalt des oeffentlichen Rechts
Hauptsitze: Stuttgart, Karlsruhe, Mannheim, Mainz
HRA 12704
Amtsgericht Stuttgart