[zeromq-dev] Publisher side filtering...

[Martin Sustrik]

Hi Gerard,
>
> I've read the mails about publisher side filtering here:
>
> http://thread.gmane.org/gmane.network.zeromq.devel/3560
>
> - Is there now a current ongoing effort to put publisher-side 
> filtering in 0MQ that I may possibly contribute to, which also allows
>   API users to specify their own methods of filtering as they see fit?
These are two distinct issues:

1. Publisher-side filtering a.k.a. subscription forwarding.
2. Custom filtering algorithms.

The former is a pretty clear functionality that has to be implemented 
sooner or later. If you want to contribute to that, you are welcome.

The latter is something that pops up every now and then but nobody have 
proposed any clear semantics for it yet (especially w.r.t. how it 
interacts with the subscription forwarding). Thus, if you want to 
contribute this kind of functionality, you have to define the intended 
semantics first.

> Background:
> ------------------
>
> I'm a researcher on Self Organizing Distributed Systems and for a 
> project, I'm trying to come up with a prototype for a (large-scale) 
> distributed infrastructure.
> This infrastructure may span multiple networks, a network may 
> join/leave or get disconnected at any time,  subscribers/publishers 
> may leave/join
> as they please (or it may just happen). New applications may get 
> deployed that use a new range of topics or require producers on 
> different networks to
> be available for their (full) function.
>
> It should become quite evident that simply publishing each produced 
> message around the network as a whole will not be acceptable, due to 
> the volume that
> eventually will be produced. There are no explicit latency or volume 
> requirements or guarantees, but of course it's good to plan for the 
> worst.
>
> I'm interested in publisher-side filtering, because it would allow for 
> a way to distribute messages selectively:
> 1. From local network broker to network edge router
> 2. From network edge router to other network edges
> 3. From router to local broker
> 4. From local broker to only those clients that are interested.

Yes. That's why it has to be implemented. There's no substitute for 
subscription forwarding when the network size exceeds certain limit.

>
> Reasons for not publishing everywhere:
> 1. Total message volume.
> 2. Additional security
> 3. Local CPU processing at each client
> 4. Extra network load per client

Yes.

NB: The two lists above make strong argument for subscription 
forwarding. However, they are irrelevant to custom filtering functionality.

> I started considering what is necessary for this to be a flexible 
> method for everyone. One possible design is the "callback" approach,
> where a filter callback method is set on a socket. Before the message 
> is sent on a single pipe/socket endpoint, the callback is called,
> the result of which determines if the send operation is executed or not.
>
> This would require changes to the "writer_t" structure to include the 
> identity and some means for the registration of the callback function.
> One complexity here is the use of an identity string having some 
> semantics (leaving messages in a queue on the publisher side) and
> multipart messages of course
> (multipart could be solved by returning the same value as for the 
> first message that popped up and then clearing this state when the 
> last message comes in)
As already said, if you want to have custom filtering algorithms, you 
have to define how the whole thing would work. How are individual 
algorithms identified? How is the algorithm forwarded to the publisher 
node? How should the publisher node behave if the subscriber nodes have 
conflicting filtering algorithms? Etc.

Martin