[zeromq-dev] pub/sub topic matching on sender side
sustrik at 250bpm.com
Fri Aug 13 16:18:26 CEST 2010
Here's a short summary of what needs to be done to get sender side
First of all it's necessary to establish bi-directional channels for
pub/sub communication rather than unidirectional as the case is now.
One direction will be used for passing messages downstream (no change to
current implemtation) the other will be used to pass subscritptions
Namely it means providing bidirectional communication in following cases:
1. Socket to I/O thread communication (pipes).
2. Socket to socket communication (pipes, inproc transport).
3. TCP connection
4. IPC connection (same as 3)
Note that multicast (PGM/EPGM) is inherently uni-directional and thus
cannot be modified this way.
Each subsription/unsubsctription has to be passed upstream. There are
several special cases:
1. If the next hop is over IPC or TCP, the peer may fail and restart. To
keep subscriptions working, receiving node has to store all the
subscriptions and resend them to the peer after reconnection.
2. At the points where subscription cannot be passed further up (either
PUB socket or multicast receiver) there has to be a matching engine,
filtering the incoming data.
3. At the points where we cannot get subscriptions from downstream
(multicast sender) we have to send all the messages.
Further, existing matching machine (prefix_tree_t) is a simple filter.
On a PUB node we have to have one such filter per outbound connection.
For performance reasons these should be merged into a single message
Note: With above algorithm, unsubscribe can be delayed. I.e. we
unsubscribe from particular message feed but still get few messages
before the unsubscription takes effect. If we believe this is an
unacceptable behaviour, there has to be a matching engine on each node.
Next, forwarder device should be modified to pass subscriptions
upstream. This can be done by creating XPUB and XSUB socket types, new
socket types that would allow user to explicitly manipulate the
subscription/unsubscription messages. That way forwarder could receive
subscriptions from XPUB and send them to XSUB.
It's not easy, but if you are not frightened and still want to give it a
try we can discuss what the best first step on the path should be.
More information about the zeromq-dev