[zeromq-dev] Publisher side filtering...

[Gerard Toonstra]

Hi all,

I've read the mails about publisher side filtering here:

http://thread.gmane.org/gmane.network.zeromq.devel/3560

- Is there now a current ongoing effort to put publisher-side filtering in
0MQ that I may possibly contribute to, which also allows
  API users to specify their own methods of filtering as they see fit?


Background:
------------------

I'm a researcher on Self Organizing Distributed Systems and for a project,
I'm trying to come up with a prototype for a (large-scale) distributed
infrastructure.
This infrastructure may span multiple networks, a network may join/leave or
get disconnected at any time,  subscribers/publishers may leave/join
as they please (or it may just happen). New applications may get deployed
that use a new range of topics or require producers on different networks to
be available for their (full) function.

It should become quite evident that simply publishing each produced message
around the network as a whole will not be acceptable, due to the volume that
eventually will be produced. There are no explicit latency or volume
requirements or guarantees, but of course it's good to plan for the worst.

I'm interested in publisher-side filtering, because it would allow for a way
to distribute messages selectively:
1. From local network broker to network edge router
2. From network edge router to other network edges
3. From router to local broker
4. From local broker to only those clients that are interested.

Reasons for not publishing everywhere:
1. Total message volume.
2. Additional security
3. Local CPU processing at each client
4. Extra network load per client


I started considering what is necessary for this to be a flexible method for
everyone. One possible design is the "callback" approach,
where a filter callback method is set on a socket. Before the message is
sent on a single pipe/socket endpoint, the callback is called,
the result of which determines if the send operation is executed or not.

This would require changes to the "writer_t" structure to include the
identity and some means for the registration of the callback function.
One complexity here is the use of an identity string having some semantics
(leaving messages in a queue on the publisher side) and
multipart messages of course
(multipart could be solved by returning the same value as for the first
message that popped up and then clearing this state when the last message
comes in)

So far, I intended to use a REQ/REP socket for clients to communicate with
brokers directly to communicate their configuration/intention
(registration on topics, topic publication, etc.). Another PUB/SUB pair is
then used to actually carry the data messages. The application / broker
memory
then maintains a subscription registry for each connected client and this
way knows what the subscriptions are.


Comments and advice is very welcome!


-- 
Gerard Toonstra
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.zeromq.org/pipermail/zeromq-dev/attachments/20101022/d69423ca/attachment.htm>