[zeromq-dev] Persistence with ZeroMQ

Pieter Hintjens ph at imatix.com
Thu Oct 16 14:54:00 CEST 2014


On Thu, Oct 16, 2014 at 1:49 PM, Holger Joukl <Holger.Joukl at lbbw.de> wrote:

> - I don't quite follow the distinction between live stream data
> and publish-subscribe. IIUC pub-sub is described as a "segmentation" of
> the live stream, i.e. when consuming a live stream I'd receive all
> data on a certain "channel" (for lack of better words) but when
> subscribing to a publication I'd only receive messages belonging
> to/carrying a certain topic/subject.

A stream is a convenient top-level isolation. Typically one stream
serves one application. Within streams, readers choose subjects they
are interested in, using pattern matching.

> - Is the concept of on-demand streamed data persistence meant to be/do
> s.th. along the lines of Apache Kafka (or https://github.com/miniway/zper), i.e.
> a per-topic/subject recorded journal of messages with a certain expiry
> period?

Not sure. It's meant to solve the late joiner and reconnection issues,
so a reader can switch between historical and live data invisibly
(except perhaps for timestamps in the message).

> Would each malamute broker record the data all its connected consumer
> clients are interested in or would the publisher's malamute broker record the data?

Yes, the broker. Writers do not see readers.

> How will be made sure that every published message will indeed get
> persisted?

There's no guarantee of this. We assume (hah!) that the broker is
reliable, and that the failures we expect and have to solve are in
clients. In practice we may use redundant brokers, for instance. To be
seen.

> When a consumer demands delivery of missed (by him) or historic messages,
> is the consumer responsible for its own delivery state and for sorting out
> duplicates?

Yes. This would be hidden in the client stack.

> - Where will the (PGM or NORM) multicast communication come into play?

As an optional output socket for any given stream. Configurable, I guess.

-Pieter



More information about the zeromq-dev mailing list