[zeromq-dev] Pub/Sub anchoring/keyframing and administrative layers in general

[Oliver Smith]

On 7/25/2010 2:53 AM, Martin Sustrik wrote:
>> While refactoring a data-streaming subsystem today, I realized that
>> using ZeroMQ with pub/sub would save me having to repeat the
>> distribution process across a whole chain of processes.
>>
>> With the exception that, at some point, the listeners need to obtain the
>> base state.
>>      
> I suspect what you need is last-value cache. I.e. if you subscribe for
> topic X you first get the latest known value of X, then catch up with
> the stream of updates as usual. Right?
>    
In this case, many Xs - but that may be an unimportant semantic.

Our project is a Massively Multi-Player online FPS wargame (that's been 
running since 2001). The system I'm re-factoring at the moment is the 
state-notification of ownerships at the town and town-sub-component 
levels. One process acts as master for this data, but then forwards it 
to the various client-facing processes which have to repackage it and 
retransmit it to their connected clients.

Before a given client can start applying changes, though, it needs the 
most recent snapshot of the overall picture.

My first-hash implementation adds a thread listening on a second socket 
REQ/REP socket, which builds an image of the current snapshot and then 
itself joins the PUB/SUB socket. This way it can apply changes to the 
snapshot as they occur, updating the msg.data() of it's pre-prepared 
message, and then simply socket.send(msg, ZMQ_NOBLOCK) back to anyone 
who requests the list. The pub/sub thread waits to get a reply from this 
thread before it starts sending updates, to avoid a race condition.

I can just see myself repeating this pattern a lot and potentially 
ending up with an army of sockets for different streams.


- Oliver