[zeromq-dev] Publisher side filtering... (draft II )

Gerard Toonstra gtoonstra at gmail.com
Sun Dec 5 18:02:44 CET 2010


thanks, this is another partial draft patch submission and this time it
certainly looks a lot cleaner than last time.
Everything implemented in the XSUB/XPUB classes. the only difference between
SUB / XSUB seems to be that SUB implements
xsetsockopt and XSUB implements the functionality to send the message
towards the session through the pipe. The difference
between PUB and XPUB is that the PUB does not receive subscription messages
in the user process, whereas XPUB does.

Comments are abound in the patch, which help to define the reasoning. I have
tested sparsely with a simple client, server
and broker (attached).

This is partial, because there is no publisher side filtering implemented
here yet. You mentioned something about not doing this per session,
but through a matching engine. Not sure how that would work.

Subscriptions do get forwarded to PUB/XPUB already, but those are not
actively used for filtering there. You mentioned John working on this,
so I hope that can be merged together.

It is assumed that setsockopt(.... ZMQ_SUBSCRIBE ) is done *after* the call
to zmq_connect. If not, there are no pipes available to write to the session
and more state must be saved in the socket_base.

Overview of how this works:

Connection topology looks like:

server implements PUB -> XSUB [ broker ] XPUB -> client implements SUB

Normal client connection:

1. Client starts a SUB socket.
2. Client connects to XPUB.
3. Client performs subscribe requests
4. The SUB socket base processes xsetsockopt requests and builds messages to
send to xsub
5. The xsub base class performs the xsend handling, decodes the data part
and updates its private subscription list (trie_t). Then sends message over
6. The XPUB session receives the request, parses it and updates its own
subscription list. Then XPUB lets message bubble up further to user process
7. The broker receives the message and does its own thing with it
8. The broker sends the message through to the XSUB socket.
9. The XSUB socket does its handling, goto 5 as long as there are XSUB
10. Eventually, a PUB socket receives the message, updates its own list, but
does not let the message bubble up to the user process.

Not yet implemented; the publisher side filtering in XPUB.  With a broker
inbetween, I'd say publisher filtering in PUB is even optional...?

Client disconnects:
1. The client is killed. The SUB socket disappears.
2. A connection in XPUB is detached. This connection has a session. The
subscription list for that session is known. For each subscription in that
   generate an "unsubscribe" message on behalf of the disconnected SUB
process and write this to the pipe. These messages go to the user process
   reading through a zmq_poll call, etc...
3. The user process receives unsubscribe requests for each previous
subscription, one by one, through the zmq_recv call.
4. The user process does its own thing.
5. The user process forwards the message through the XSUB socket.

Broker disconnects:
1. The broker dies. The XPUB and XSUB sockets disappear.
2. Any XPUB that was connected to XSUB will start generating unsubscription
3. Both PUB and XPUB throw away the subscription list (since clients will
resubscribe on connection).
4. A broker restart means the lists are all empty.
5. Clients reconnect, resend their subscriptions to the broker.
6. SUB client reconnects and PUB clients reconnects happen at the same time.
7. On a reconnect to the XSUB, the XSUB forwards all subscriptions it knows
about at that time.
8. Any new changes to subscriptions are handled like the first use case.

Let me know about any changes that still need to be made. I have not
thoroughly tested this in a real-life environment.



On Sat, Dec 4, 2010 at 11:17 PM, Martin Sustrik <sustrik at 250bpm.com> wrote:

> Gerard,
>  I would love to help with this effort. I don't have enough free time to
>> do all of it, but I can at least put some infrastructure pieces in
>> place. Say I can separate XPUB/XSUB from PUB/SUB.
> Ok. Done. There's XPUB and XSUB in the master. They behave exactly the same
> as PUB and SUB at the moment.
> Martin

Gerard Toonstra
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.zeromq.org/pipermail/zeromq-dev/attachments/20101205/4497ce6f/attachment.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: main.c
Type: text/x-csrc
Size: 2737 bytes
Desc: not available
URL: <https://lists.zeromq.org/pipermail/zeromq-dev/attachments/20101205/4497ce6f/attachment.c>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: diff.patch
Type: text/x-patch
Size: 16018 bytes
Desc: not available
URL: <https://lists.zeromq.org/pipermail/zeromq-dev/attachments/20101205/4497ce6f/attachment.bin>

More information about the zeromq-dev mailing list