[zeromq-dev] durable subscirbers and inproc

[Michael Kogan]

I don't need really guaranteed delivery, what I need is to be able to know if I am missing messages. The messages in my case represent data that is durably stored elsewhere, so I don't need a durable message store ala AMQP.
The per-publisher sequence numbers are reasonable in fact that is the best I could come up with as well.
Still, if subscribers are not subscribing to all messages, but using the first message of multipart messages to route (as in the example in the guide, which I am using, thank you), the messages received will not be free of sequence gaps. Of course, if we know that we have a reasonably constrained set of theses discrete routing messages (I called them channels), we could number along them too. I think it may be a neat feature for 0mq to have that kind of scheme on top of regular pub/sub so that the publisher socket would use its id and routing "header" information to maintain sequences and subscriber socket would know if it is missing messages.

Mike.

P.S. Thanks to all that are so responsive on the list and for developing a really neat piece of software.


On Feb 23, 2011, at 11:13 AM, John Flanagan wrote:

> 
> 
> On Wed, Feb 23, 2011 at 12:09 PM, Chuck Remes <cremes.devlist at mac.com> wrote:
> 
> On Feb 23, 2011, at 11:31 AM, Pieter Hintjens wrote:
> 
> > On Wed, Feb 23, 2011 at 6:25 PM, Chuck Remes <cremes.devlist at mac.com> wrote:
> >
> >> If you really need reliable delivery of each published message, then the Pub/Sub pattern is not appropriate. You should use the REQ/REP pattern because you'll need to acknowledge each message.
> >
> > Even that won't guarantee delivery :-) What if your acknowledgment gets lost?
> 
> True. Then you need timeouts and retransmissions. Definitely complicated stuff...
> 
> > Plus it adds extraordinary latency to traffic. Depending on your
> > application, you would want some kind of asynchronous negative
> > acknowledgment, i.e. "I'm missing data X, please (re)send". Possibly
> > out of band so the normal pubsub traffic can scale to many clients
> > over multicast.
> 
> Batching up the missing sequence numbers can be difficult. I like using a "sliding windows" algo for acknowledgement. You essentially end up recreating the sliding windows logic from zmodem, tcp, etc.
> 
> This, this, this.
> 
> By the time you layer on all the things you need to implement to achieve reasonable reliability guarantees, you will have IMPLEMENTED tcp... poorly.
> 
> If you need reliable messaging, you need tcp.  The only way around this is to NOT need reliable messaging.  If there's no possible way to avoid reliable messaging, then get used to using tcp.  Seriously.  You will save yourself endless heartache.
> 
> The interesting discussion then becomes figuring out ways to avoid or relax the reliability requirement.  That's a complex topic and the tradeoff choices tend to get made for domain-specific reasons so the solutions frequently fail to be useful in the generic case.
> _______________________________________________
> zeromq-dev mailing list
> zeromq-dev at lists.zeromq.org
> http://lists.zeromq.org/mailman/listinfo/zeromq-dev

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.zeromq.org/pipermail/zeromq-dev/attachments/20110223/9ac8311f/attachment.htm>