[zeromq-dev] Pub/Sub anchoring/keyframing and administrative layers in general

[Oliver Smith]

On 7/25/2010 3:25 AM, Pieter Hintjens wrote:
> The definition of "base state" depends on the use case, presumably.
>
> For market data, which consists of values for specific "instruments"
> (e.g. stocks), the base state is the last known price for an
> instrument.  So when subscribing to a topic tree, you would want a
> snapshot of all instruments in that tree.
That's a good analogy for what I'm trying to do.
>    For video, it's a key frame.
>
> The problem with doing this in-band is that it messes up delivery of
> current data, i.e. you'll get an arbitrary lag at startup.  The
> problem doing this out-of-band is synchronization.  For market data
> it's clear how to do that: you simply throw out snapshot data for
> which you've already received a more recent update, and keep what's
> left.  For video data... how can you use an out of date keyframe?
>    

That's kind of the issue I was pondering, but I actually thought that 
they were more similar. In both cases it's just a snapshot at a certain 
point.

In the Market Data example, you would update the entry for instruments 
in your cache as updates came past. In the KeyFrame example, you might 
replace the keyframe with each subsequent keyframe sent, and then modify 
the keyframe cache with each update as it arrives.

In both cases, you update the cache before sending updates. Doing the 
cache in-band means that when you first connect, you won't receive any 
updates until you are sent the initial snapshot simply because, at the 
lowest level, zmq would do the equivalent of: accept() && sendSnapshot() 
&& addToLocalSubscriberList().

Because my dataset is market-data like, my methodology is just going to 
be to have each client connect first to the PUB/SUB socket, then fetch 
the latest snapshot from the REQ/REP socket, and then begin recv()ing 
from the PUB/SUB socket, so that it may receive old updates on the 
pub/sub connection but it won't miss any that occured between it 
requesting the snapshot and beginning recv()ing the SUB socket.

- Oliver