[zeromq-dev] Persistence with ZeroMQ

Holger Joukl Holger.Joukl at LBBW.de
Thu Oct 16 13:49:36 CEST 2014


zeromq-dev-bounces at lists.zeromq.org schrieb am 13.10.2014 15:36:38:

> Von: Pieter Hintjens
> Mohit,
> I've started on a new broker project,
> https://github.com/Malamute/malamute, which aims to do what you're
> asking for.
> If you would like to read the description and tell me how closely it
> matches, that would be helpful.

I've some questions wrt the interesting malamute whitepaper:

- I don't quite follow the distinction between live stream data
and publish-subscribe. IIUC pub-sub is described as a "segmentation" of
the live stream, i.e. when consuming a live stream I'd receive all
data on a certain "channel" (for lack of better words) but when
subscribing to a publication I'd only receive messages belonging
a certain topic/subject.

The channel probably being an address, i.e. basically a network protocol
+ port combo.

But isn't receiving live stream data just pub-sub as well, basically using
a wildcard topic/subject?

- Is the concept of on-demand streamed data persistence meant to be/do
along the lines of Apache Kafka (or https://github.com/miniway/zper), i.e.
per-topic/subject recorded journal of messages with a certain expiry

Would each malamute broker record the data all its connected consumer
are interested in or would the publisher's malamute broker record the data?

How will be made sure that every published message will indeed get

When a consumer demands delivery of missed (by him) or historic messages,
is the
consumer responsible for its own delivery state and for sorting out

- Where will the (PGM or NORM) multicast communication come into play?
I assume
 * (sending) client connects to a malamute broker (IPC to a local or TCP to
a remote broker)
 * (sending) client publishes messages (sends data to a stream), or rather
   the messages to its broker
 * sender's broker distributes the messages to other malamute brokers
 * those deliver the messages to their connected consumer clients.

The whitepaper states that in a distributed topology with one malamute
per host those will communicate via TCP.

Best regards,

Landesbank Baden-Wuerttemberg
Anstalt des oeffentlichen Rechts
Hauptsitze: Stuttgart, Karlsruhe, Mannheim, Mainz
HRA 12704
Amtsgericht Stuttgart

More information about the zeromq-dev mailing list