[zeromq-dev] pub/sub topic matching on sender side

[Jon Dyte]

Martin Sustrik wrote:

Thanks Martin this is most useful.

It's quite complex and I can see the core would need changes.

If initially only the inproc was tackled , is it in anyway simpler, in 
yr opinion?

jon

> Hi Jon,
>
> Here's a short summary of what needs to be done to get sender side 
> subscription matching.
>
> First of all it's necessary to establish bi-directional channels for 
> pub/sub communication rather than unidirectional as the case is now.
>
> One direction will be used for passing messages downstream (no change to 
> current implemtation) the other will be used to pass subscritptions 
> upstream.
>
> Namely it means providing bidirectional communication in following cases:
>
> 1. Socket to I/O thread communication (pipes).
> 2. Socket to socket communication (pipes, inproc transport).
> 3. TCP connection
> 4. IPC connection (same as 3)
>
> Note that multicast (PGM/EPGM) is inherently uni-directional and thus 
> cannot be modified this way.
>
> Each subsription/unsubsctription has to be passed upstream. There are 
> several special cases:
>
> 1. If the next hop is over IPC or TCP, the peer may fail and restart. To 
> keep subscriptions working, receiving node has to store all the 
> subscriptions and resend them to the peer after reconnection.
>
> 2. At the points where subscription cannot be passed further up (either 
> PUB socket or multicast receiver) there has to be a matching engine, 
> filtering the incoming data.
>
> 3. At the points where we cannot get subscriptions from downstream 
> (multicast sender) we have to send all the messages.
>
> Further, existing matching machine (prefix_tree_t) is a simple filter. 
> On a PUB node we have to have one such filter per outbound connection. 
> For performance reasons these should be merged into a single message 
> matching mechanism.
>
> Note: With above algorithm, unsubscribe can be delayed. I.e. we 
> unsubscribe from particular message feed but still get few messages 
> before the unsubscription takes effect. If we believe this is an 
> unacceptable behaviour, there has to be a matching engine on each node.
>
> Next, forwarder device should be modified to pass subscriptions 
> upstream. This can be done by creating XPUB and XSUB socket types, new 
> socket types that would allow user to explicitly manipulate the 
> subscription/unsubscription messages. That way forwarder could receive 
> subscriptions from XPUB and send them to XSUB.
>
> It's not easy, but if you are not frightened and still want to give it a 
> try we can discuss what the best first step on the path should be.
>
> Martin
> _______________________________________________
> zeromq-dev mailing list
> zeromq-dev at lists.zeromq.org
> http://lists.zeromq.org/mailman/listinfo/zeromq-dev
>
>