[zeromq-dev] pub/sub topic matching on sender side

[Jon Dyte]

George Coles wrote:
> Perhaps you could get by with creating a device which is specialized to do subscription filtering? And have clients connect to it as a proxy? That way you could keep the core untouched.
>
>   


yes george, it can be done that way as well.

in fact Matt W posted this diagram the other day suggesting I think the same
which looked like this

[[[ publisher ]]] >>>   [[[ filter ]]] >>>  [[[ pub socket ]]]
			     ^^^<<<	[[[ subscription updates ]]] <<< [[[ sub socket ]]]

so i think in this scheme the subscriber sends out topic messages up the 
stream with its along with some uuid. the the filter applies the topic
then it's published to the subcriber after looking up which socket is 
associated with the uuid, or just over pub/sub, though that might involve
mutiple publishes as far as I can see.

> -----Original Message-----
> From: zeromq-dev-bounces at lists.zeromq.org [mailto:zeromq-dev-bounces at lists.zeromq.org] On Behalf Of Martin Sustrik
> Sent: Friday, August 13, 2010 10:18 AM
> To: 0MQ development list
> Subject: Re: [zeromq-dev] pub/sub topic matching on sender side
>
> Hi Jon,
>
> Here's a short summary of what needs to be done to get sender side 
> subscription matching.
>
> First of all it's necessary to establish bi-directional channels for 
> pub/sub communication rather than unidirectional as the case is now.
>
> One direction will be used for passing messages downstream (no change to 
> current implemtation) the other will be used to pass subscritptions 
> upstream.
>
> Namely it means providing bidirectional communication in following cases:
>
> 1. Socket to I/O thread communication (pipes).
> 2. Socket to socket communication (pipes, inproc transport).
> 3. TCP connection
> 4. IPC connection (same as 3)
>
> Note that multicast (PGM/EPGM) is inherently uni-directional and thus 
> cannot be modified this way.
>
> Each subsription/unsubsctription has to be passed upstream. There are 
> several special cases:
>
> 1. If the next hop is over IPC or TCP, the peer may fail and restart. To 
> keep subscriptions working, receiving node has to store all the 
> subscriptions and resend them to the peer after reconnection.
>
> 2. At the points where subscription cannot be passed further up (either 
> PUB socket or multicast receiver) there has to be a matching engine, 
> filtering the incoming data.
>
> 3. At the points where we cannot get subscriptions from downstream 
> (multicast sender) we have to send all the messages.
>
> Further, existing matching machine (prefix_tree_t) is a simple filter. 
> On a PUB node we have to have one such filter per outbound connection. 
> For performance reasons these should be merged into a single message 
> matching mechanism.
>
> Note: With above algorithm, unsubscribe can be delayed. I.e. we 
> unsubscribe from particular message feed but still get few messages 
> before the unsubscription takes effect. If we believe this is an 
> unacceptable behaviour, there has to be a matching engine on each node.
>
> Next, forwarder device should be modified to pass subscriptions 
> upstream. This can be done by creating XPUB and XSUB socket types, new 
> socket types that would allow user to explicitly manipulate the 
> subscription/unsubscription messages. That way forwarder could receive 
> subscriptions from XPUB and send them to XSUB.
>
> It's not easy, but if you are not frightened and still want to give it a 
> try we can discuss what the best first step on the path should be.
>
> Martin
> _______________________________________________
> zeromq-dev mailing list
> zeromq-dev at lists.zeromq.org
> http://lists.zeromq.org/mailman/listinfo/zeromq-dev
> _______________________________________________
> zeromq-dev mailing list
> zeromq-dev at lists.zeromq.org
> http://lists.zeromq.org/mailman/listinfo/zeromq-dev
>
>