[zeromq-dev] Publisher side filtering...

[Gerard Toonstra]

Hi Martin,

On Fri, Oct 29, 2010 at 7:00 PM, Martin Sustrik <sustrik at 250bpm.com> wrote:

> Hi Gerard,
>
> 1. Existing code must not break because of changes ( the default
>> initialization reflects how people use it today ). This means default it
>> does not use filtering at all.
>>
>
> There's filtering there even today. Have you see ZMQ_SUBSCRIBE socket
> option?


Yes, I meant here that publisher sockets don't do any fitlering at all.


> I think you've started too high in the stack.
>

> Note that there's a "subscription" feature in 0MQ even today. We don't
> really have to change semantics or add any new concepts to the design. All
> that's needed is to forward the subscriptions from the subscriber all the
> way up through the distribution hierarchy.
>
> That consists of 3 more or less self-contained parts:
>
> 1. Forwarding of subscription from the application thread (socket) to the
> I/O thread (session).
>

this could essentially be the same setsockopt call?


> 2. Forwarding of subscription from the I/O thread to the peer application.
> In other words, the subscription has to be passed on the wire upstream.
>

If this is over the same connection that data is received from through the
pub socket, then all protocols must support
reads and writes happening at the same time without any mechanism for
synchronization (request/reply, etc). Could this become a problem?
The benefit of doing this is that there's no more need to find a connection
through an identity and no further need for an additional socket to be
created.

3. Forwarding a subscription through a forwarder device.
>

It says in the manual that forwarding devices should only use PUB/SUB
sockets. There is only one in socket (not a list of sockets) used in the API
call.
This implies that the forwarding device is getting all messages from a
single publisher.

The forwarding device has a number of connections to clients towards it is
publishing messages. The PUB socket is the component that does the
filtering.
This means that the forwarder should have a means to query the PUB socket
for subscriptions that are currently active, union them together and then
send the union of that upstream to the single publisher it is connected to.

If there is any one client on the forwarder that has not sent a subscription
upstream, then the result of such a union is *, or no filter at all.



While accomplishing these three steps there are several problems to solve:
>
> 1. The filtering algorithm has to be able to maintain several sets of
> filters (one per downstream connection). Jon Dyte have done some work on
> this. He may share the code.
>

If Jon is reading this, is there any way in which we can collaborate?


>
> 2. Unsubscriptions have to be propagated up the tree in the same way as
> subscriptions.
>

ZMQ_UNSUBSCRIBE / setsockopt.


>
> 3. When connection between two nodes breaks, related unsubscriptions should
> be generated automatically.
>

I found this discussion on the list related to this whole thing:

http://www.mail-archive.com/zeromq-dev@lists.zeromq.org/msg03420.html

Does it actually matter if an unsubscribe doesn't take place directly
upstream?   I think it matters for the
client process, but the client has been subscribed and there has been a flow
of traffic before. So if you want
immediate reduction of traffic flow based on an UNSUBSCRIBE, I'd look into
generating the unsubscribes based on
disconnects etc...

One potential problem here is that state at either side of the connection
may start to diverge over longer periods of time
due to whatever bug, inconsistency or network failure.

Another approach here is to periodically send the full subscription list
over the wire and allow, for a short time period,
the waste in network traffic.

The benefit of doing this is that correct state is better guaranteed and
there is no need to hunt down potential unsubscribes,
because the list of currently known active subscriptions is being
transmitted at some point in time.



> 4. When connection between two nodes in (re)established, subscriber should
> forward all the subscriptions upstream.
>
> Martin
>

individual calls to ZMQ_SUBSCRIBE, or should this be optimized using e.g. a
delimiter or an array? (ZMQ_SUBSCRIBE_LIST).


Comments welcome,

rgds,

G>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.zeromq.org/pipermail/zeromq-dev/attachments/20101030/86cd620b/attachment.htm>