[zeromq-dev] durable subscirbers and inproc

Pieter Hintjens ph at imatix.com
Thu Feb 24 00:39:39 CET 2011


Hi Michael,

You're absolutely right to say that sequencing won't work if the
subscriber uses a filter, which most will.

Your concept of channel matches the stream concept I've explained
here: http://zeromq.org/osdp, that may give you some more ideas. It's
a product we hope to build over time, with input from clients and
users.

Regards
-
Pieter Hintjens
iMatix

On Wed, Feb 23, 2011 at 10:24 PM, Michael Kogan
<mkogan at semanticresearch.com> wrote:
> I don't need really guaranteed delivery, what I need is to be able to know
> if I am missing messages. The messages in my case represent data that is
> durably stored elsewhere, so I don't need a durable message store ala AMQP.
> The per-publisher sequence numbers are reasonable in fact that is the best I
> could come up with as well.
> Still, if subscribers are not subscribing to all messages, but using the
> first message of multipart messages to route (as in the example in the
> guide, which I am using, thank you), the messages received will not be free
> of sequence gaps. Of course, if we know that we have a reasonably
> constrained set of theses discrete routing messages (I called them
> channels), we could number along them too. I think it may be a neat feature
> for 0mq to have that kind of scheme on top of regular pub/sub so that the
> publisher socket would use its id and routing "header" information to
> maintain sequences and subscriber socket would know if it is missing
> messages.
> Mike.
> P.S. Thanks to all that are so responsive on the list and for developing a
> really neat piece of software.
>
> On Feb 23, 2011, at 11:13 AM, John Flanagan wrote:
>
>
> On Wed, Feb 23, 2011 at 12:09 PM, Chuck Remes <cremes.devlist at mac.com>
> wrote:
>>
>> On Feb 23, 2011, at 11:31 AM, Pieter Hintjens wrote:
>>
>> > On Wed, Feb 23, 2011 at 6:25 PM, Chuck Remes <cremes.devlist at mac.com>
>> > wrote:
>> >
>> >> If you really need reliable delivery of each published message, then
>> >> the Pub/Sub pattern is not appropriate. You should use the REQ/REP pattern
>> >> because you'll need to acknowledge each message.
>> >
>> > Even that won't guarantee delivery :-) What if your acknowledgment gets
>> > lost?
>>
>> True. Then you need timeouts and retransmissions. Definitely complicated
>> stuff...
>>
>> > Plus it adds extraordinary latency to traffic. Depending on your
>> > application, you would want some kind of asynchronous negative
>> > acknowledgment, i.e. "I'm missing data X, please (re)send". Possibly
>> > out of band so the normal pubsub traffic can scale to many clients
>> > over multicast.
>>
>> Batching up the missing sequence numbers can be difficult. I like using a
>> "sliding windows" algo for acknowledgement. You essentially end up
>> recreating the sliding windows logic from zmodem, tcp, etc.
>
> This, this, this.
>
> By the time you layer on all the things you need to implement to achieve
> reasonable reliability guarantees, you will have IMPLEMENTED tcp... poorly.
>
> If you need reliable messaging, you need tcp.  The only way around this is
> to NOT need reliable messaging.  If there's no possible way to avoid
> reliable messaging, then get used to using tcp.  Seriously.  You will save
> yourself endless heartache.
>
> The interesting discussion then becomes figuring out ways to avoid or relax
> the reliability requirement.  That's a complex topic and the tradeoff
> choices tend to get made for domain-specific reasons so the solutions
> frequently fail to be useful in the generic case.
> _______________________________________________
> zeromq-dev mailing list
> zeromq-dev at lists.zeromq.org
> http://lists.zeromq.org/mailman/listinfo/zeromq-dev
>
>
> _______________________________________________
> zeromq-dev mailing list
> zeromq-dev at lists.zeromq.org
> http://lists.zeromq.org/mailman/listinfo/zeromq-dev
>
>



More information about the zeromq-dev mailing list