[zeromq-dev] Persistence with ZeroMQ

[Pieter Hintjens]

On Thu, Oct 16, 2014 at 4:04 PM, Holger Joukl <Holger.Joukl at lbbw.de> wrote:

> Ok. Still more or less the same to me: A receiver subscribes to a stream
> or a subset of the stream.

A publisher produces one stream; a receiver can get N streams.

> I guess it then depends on what you'd describe as
> a pub-sub "channel",

That is the stream, yes.

> So technically "streams" would then be discriminated by ports or you'd need
> to have some kind or root topic/subject.

Client specifies them at connection time (see the protocol wireframe)

https://github.com/hintjens/malamute-core/blob/master/src/mlm_msg.bnf

> I think my wording was a bit vague here: Suppose a topology of one malamute
> broker per host and each client connecting to its local broker. Sender
> and receivers are all on different hosts.

> Will the sender's broker or the (multiple) receivers' broker(s) store
> the messages?

I assume we'll store the messages redundantly in that case, across all
brokers that share a stream. I think also the unit of sharing will be
an entire stream, not part of it, so that readers can re-scan streams
later.

> Isn't it entirely possibly that the sending client works just fine but
> receiving and/or storing the data fails?

In the broker? That would be signaled as a critical system failure,
there is nothing the sender can do to recover or buffer (imagine you
are reading NASDAQ).

> I think e.g. Apache Kafka uses an optional acknowledgement for producers.

Perhaps, I'm not against it as such, it's just not a failure I've seen
in practice. I am happy to be proven wrong.

> I take it that this would then be functionality in a malamute client
> library that reveivers link/import/include.

Yes.

>> As an optional output socket for any given stream. Configurable, I guess.
>
> I still don't get it. Would the clients listen to a multicast address
> where a broker sends out data? Or would the multicast communication be
> inter-brokers, the clients then getting the data from their brokers
> over IPC or TCP?

The first, I assume. Honestly, we're too early for this detailed
question/answer. I suggest you try the software, which already works
in basic form, and ask questions based on that rather than theoretical
reading of the whitepaper.

-Pieter