[zeromq-dev] malamut matching engine & disappearing subscription

[Pieter Hintjens]

The mlm_stream_simple engine doesn't do the bitmap magic. It uses
regexps and matches every message against every pattern, so it's
simple (hence the name) yet O(n2) slow.

The bitmap design comes from OpenAMQ and it proven, and fast. The
trouble is it's quite delicate machinery (so takes time) and I'm
waiting for an opportunity to make it. It recalculates matches as it
gets new message topics, and new/canceled subscriptions, which
introduces latency peaks. However these are minimal in practice since
the sets of topics and subscriptions are stable over time. We can in
theory cache these to reduce runtime recalculation. In OpenAMQ
deployments we never had to do this.

-Pieter

On Wed, Aug 26, 2015 at 11:42 AM, Benoit Chesneau <bchesneau at gmail.com> wrote:
> hi all,
>
> I was reading the malamut white paper about the matching engine, and I am
> trying to understand what I miss in using an inverted bitmap index to match
> consumers subscriptions. I tried to look at the source code but it seems
> that zbits is not used right now.
>
> From what I understand, bitmaps key are the subscription fields with their
> value (ex. `market=forex`) and the bitmap  allows to match the subscriptions
> (each subscription item is a bit set to 1 for a field). If I read the given
> example, the size of the bitmap equals to the number of subscriptions.
>
> My question is, what happen when a subscription is closed? Are all the
> bitmaps for the fields recalculated ? Or just the bit removed for all the
> fields of the subscription? Is the bitmap size dynamic? Or did I miss
> something?
>
> In that case is this designed for the case where we can have a lot of
> subscription created and removed over the time?
>
>
> - benoît
>
> _______________________________________________
> zeromq-dev mailing list
> zeromq-dev at lists.zeromq.org
> http://lists.zeromq.org/mailman/listinfo/zeromq-dev
>