[zeromq-dev] pubsub subscription acknowledgement

[Justin Karneges]

Hi folks,

Many "realtime" applications have a need to display past content as well 
as future changes to that content. For example, a news feed application 
might subscribe for news updates and then separately query for the most 
recent 20 news items for initial display. The order of operations is 
important. The subscription is established first so that there isn't a 
chance of losing data. In the worst case, data might be received twice 
(via the subscription and the history query), but the receiver can 
de-dup as necessary.

The 0MQ guide has some strong opinions about how pubsub should be used:
   http://zguide.zeromq.org/page:all#Pros-and-Cons-of-Pub-Sub
, and I'm not sure how to implement the pattern I described above while 
at the same time following the guide in its purest form.

Naively, one could first set up a SUB and then do a REQ to query for 
past data, but the problem is that the application needs a way of 
knowing when the SUB socket is ready before performing the query. This 
problem is further complicated if the pubsub layer is tiered. The 
subscription shouldn't be considered ready until has gone all the way 
upstream.

Here are a few ideas I've been thinking about:

1) If the transport allows bidirectional communication (e.g. TCP, but 
not PGM), then XPUB could be used on the publisher side to send down an 
ack whenever a subscription request arrives. If there are tiers, wait 
for upstream acks before sending acks downstream.

2) If PGM is used, the publisher should send heartbeats regularly. 
Receipt of any message means the subscription is functional. For this to 
work in a tiered architecture, the topmost publisher would need to send 
the heartbeats. However, this only works if the number of different 
subscriptions is minimal otherwise that's a lot of heartbeats.

3) Just sleep. Yeah, the guide considers this a hack but perhaps it is 
the most practical approach if you have tiers, pgm, and an unbounded 
number of subscription types. Regularly calculate network travel time 
between subscriber and topmost publisher, then sleep for however long 
that value is and call it done.

Maybe there are other ideas? Thanks for any tips.

Justin