[zeromq-dev] durable subscirbers and inproc

Michael Kogan mkogan at semanticresearch.com
Thu Feb 24 01:14:29 CET 2011


Ooops, finger slipped.
Pieter,  what is the difference between the "stream" you describe in your document and forwarder, other than use of a registration server? Do the streams in your estimation have a location? Or are they a collection of connected forwarders (one per box)?

Thank you,

Mike.

On Feb 23, 2011, at 4:03 PM, Michael Kogan wrote:

> So are you  saying I can't have this now? :) This sounds like a generalization of the approach I have been thinking about, using ZK for coordination.
> 
> On Feb 23, 2011, at 3:39 PM, Pieter Hintjens wrote:
> 
>> Hi Michael,
>> 
>> You're absolutely right to say that sequencing won't work if the
>> subscriber uses a filter, which most will.
>> 
>> Your concept of channel matches the stream concept I've explained
>> here: http://zeromq.org/osdp, that may give you some more ideas. It's
>> a product we hope to build over time, with input from clients and
>> users.
>> 
>> Regards
>> -
>> Pieter Hintjens
>> iMatix
>> 
>> On Wed, Feb 23, 2011 at 10:24 PM, Michael Kogan
>> <mkogan at semanticresearch.com> wrote:
>>> I don't need really guaranteed delivery, what I need is to be able to know
>>> if I am missing messages. The messages in my case represent data that is
>>> durably stored elsewhere, so I don't need a durable message store ala AMQP.
>>> The per-publisher sequence numbers are reasonable in fact that is the best I
>>> could come up with as well.
>>> Still, if subscribers are not subscribing to all messages, but using the
>>> first message of multipart messages to route (as in the example in the
>>> guide, which I am using, thank you), the messages received will not be free
>>> of sequence gaps. Of course, if we know that we have a reasonably
>>> constrained set of theses discrete routing messages (I called them
>>> channels), we could number along them too. I think it may be a neat feature
>>> for 0mq to have that kind of scheme on top of regular pub/sub so that the
>>> publisher socket would use its id and routing "header" information to
>>> maintain sequences and subscriber socket would know if it is missing
>>> messages.
>>> Mike.
>>> P.S. Thanks to all that are so responsive on the list and for developing a
>>> really neat piece of software.
>>> 
>>> On Feb 23, 2011, at 11:13 AM, John Flanagan wrote:
>>> 
>>> 
>>> On Wed, Feb 23, 2011 at 12:09 PM, Chuck Remes <cremes.devlist at mac.com>
>>> wrote:
>>>> 
>>>> On Feb 23, 2011, at 11:31 AM, Pieter Hintjens wrote:
>>>> 
>>>>> On Wed, Feb 23, 2011 at 6:25 PM, Chuck Remes <cremes.devlist at mac.com>
>>>>> wrote:
>>>>> 
>>>>>> If you really need reliable delivery of each published message, then
>>>>>> the Pub/Sub pattern is not appropriate. You should use the REQ/REP pattern
>>>>>> because you'll need to acknowledge each message.
>>>>> 
>>>>> Even that won't guarantee delivery :-) What if your acknowledgment gets
>>>>> lost?
>>>> 
>>>> True. Then you need timeouts and retransmissions. Definitely complicated
>>>> stuff...
>>>> 
>>>>> Plus it adds extraordinary latency to traffic. Depending on your
>>>>> application, you would want some kind of asynchronous negative
>>>>> acknowledgment, i.e. "I'm missing data X, please (re)send". Possibly
>>>>> out of band so the normal pubsub traffic can scale to many clients
>>>>> over multicast.
>>>> 
>>>> Batching up the missing sequence numbers can be difficult. I like using a
>>>> "sliding windows" algo for acknowledgement. You essentially end up
>>>> recreating the sliding windows logic from zmodem, tcp, etc.
>>> 
>>> This, this, this.
>>> 
>>> By the time you layer on all the things you need to implement to achieve
>>> reasonable reliability guarantees, you will have IMPLEMENTED tcp... poorly.
>>> 
>>> If you need reliable messaging, you need tcp.  The only way around this is
>>> to NOT need reliable messaging.  If there's no possible way to avoid
>>> reliable messaging, then get used to using tcp.  Seriously.  You will save
>>> yourself endless heartache.
>>> 
>>> The interesting discussion then becomes figuring out ways to avoid or relax
>>> the reliability requirement.  That's a complex topic and the tradeoff
>>> choices tend to get made for domain-specific reasons so the solutions
>>> frequently fail to be useful in the generic case.
>>> _______________________________________________
>>> zeromq-dev mailing list
>>> zeromq-dev at lists.zeromq.org
>>> http://lists.zeromq.org/mailman/listinfo/zeromq-dev
>>> 
>>> 
>>> _______________________________________________
>>> zeromq-dev mailing list
>>> zeromq-dev at lists.zeromq.org
>>> http://lists.zeromq.org/mailman/listinfo/zeromq-dev
>>> 
>>> 
>> _______________________________________________
>> zeromq-dev mailing list
>> zeromq-dev at lists.zeromq.org
>> http://lists.zeromq.org/mailman/listinfo/zeromq-dev
> 
> _______________________________________________
> zeromq-dev mailing list
> zeromq-dev at lists.zeromq.org
> http://lists.zeromq.org/mailman/listinfo/zeromq-dev




More information about the zeromq-dev mailing list