[zeromq-dev] Check if message is dropped due to HWM at send in ZeroMQ PUB-SUB pattern

Pieter Hintjens ph at imatix.com
Thu Oct 2 14:21:08 CEST 2014


By magic coincidence I'm sketching out a proposal for reliable pub-sub.

https://github.com/zeromq/zeps



On Thu, Oct 2, 2014 at 12:25 PM, Doron Somech <somdoron at gmail.com> wrote:
> So in that case what you want is Majordomo pattern or Freelance pattern if
> you don't want a broker.
>
> I don't think pubsub is the correct pattern here.
>
> On Thu, Oct 2, 2014 at 6:23 AM, Ranganath Gunawardane
> <ranganath at outlook.com> wrote:
>>
>> > Dealer-router doesn't mean request-reply, you can still use one way
>> > communication.
>> Hmm. I will again have to look at the manual.
>>
>> > Does the message go to one recipient or more?
>> A given message is only for a single recipient. Right now in pub-sub it is
>> handled by adding
>> a unique subscription per recipient.
>>
>> ________________________________
>> Date: Wed, 1 Oct 2014 13:59:44 +0300
>> From: somdoron at gmail.com
>>
>> To: zeromq-dev at lists.zeromq.org
>> Subject: Re: [zeromq-dev] Check if message is dropped due to HWM at send
>> in ZeroMQ PUB-SUB pattern
>>
>> Dealer-router doesn't mean request-reply, you can still use one way
>> communication.
>>
>> You do have to write more logic in the router to route the messages but it
>> should be pretty simple logic.
>>
>> Your protocol between the router and the dealer should include
>> registration for tags.
>>
>> I think you want something that is very similar to Majordomo
>> (http://rfc.zeromq.org/spec:7).
>>
>> Does the message go to one recipient or more?
>>
>> On Tue, Sep 30, 2014 at 2:32 PM, Ranganath Gunawardane
>> <ranganath at outlook.com> wrote:
>>
>> > Then do not use pub-sub but router-dealer.
>> Well, my user case leans more towards a pub-sub pattern than req-reply.
>> Let me explain.
>>
>> I am implementing a messaging service (a message bus) for multiple client
>> processes to talk to each other. Every process sends messages with a
>> particular 'to' tag. Also, every process subscribes to a unique tag to
>> receive messages. This messaging service does not have a sense about the
>> content of the message to implement a req-reply mechanism; that is upto the
>> clients to handle at a 'business' level.
>>
>> Sending process do not need to worry about the receive getting it. But I
>> at least need to tell the sending client whether they were successful in
>> submitting the message to the messaging layer (without hitting send HWM
>> etc.).
>>
>> Because req-reply is out of question (a slow receiver cannot bog down the
>> sender), the pub-sub seemed a better approximation.
>>
>>
>> > From: ph at imatix.com
>> > Date: Tue, 30 Sep 2014 08:41:56 +0200
>> > To: zeromq-dev at lists.zeromq.org
>> > Subject: Re: [zeromq-dev] Check if message is dropped due to HWM at send
>> > in ZeroMQ PUB-SUB pattern
>>
>> >
>> > > I want to let my publisher know if he is dropping messages because he
>> > > has hit the HWM.
>> >
>> > Then do not use pub-sub but router-dealer. Pub-sub is designed for
>> > high volumes with many subscribers. There is no useful way to deal, in
>> > the publisher, with a single slow or blocked subscriber, and ZeroMQ
>> > pub-sub does not support that.
>> >
>> > On Tue, Sep 30, 2014 at 1:36 AM, Ranganath Gunawardane
>> > <ranganath at outlook.com> wrote:
>> > > Hi All, I have implemented a message router using zmq, czmq with
>> > > pub-sub
>> > > pattern.
>> > >
>> > > In my simple test set up, I am using a publisher-subscriber with a
>> > > proxy. I
>> > > have a fast sender and a very slow receiver causing messages to hit
>> > > HWM and
>> > > drop on send. My exception is that send would fail with 'message
>> > > dropped'
>> > > error, but it is not the case. the zmq_msg_send() is not giving me any
>> > > error
>> > > even though the messages get dropped (I can verify this by seeing gaps
>> > > in
>> > > messages in subscriber end).
>> > >
>> > > I have looked into fixing slow subscriber problem by adding a sequence
>> > > number as you suggested, specially suicidal snail pattern in the
>> > > guide. But
>> > > my use case is a bit different. I really do not care about the
>> > > subscribers;
>> > > I want to let my publisher know if he is dropping messages because he
>> > > has
>> > > hit the HWM.
>> > >
>> > > I gather that this is the nature of pub-sub. At the same time, from
>> > > documentation, it says, if I use ZMQ_DONTWAIT, and 'if the message
>> > > cannot be
>> > > queued on the socket, function call shall fail'. Does it mean that I
>> > > can use
>> > > this failure as a 'send failure due to buffer full'?
>> > >
>> > > Regards.
>> > > Ranganath
>> > >
>> > > _______________________________________________
>> > > zeromq-dev mailing list
>> > > zeromq-dev at lists.zeromq.org
>> > > http://lists.zeromq.org/mailman/listinfo/zeromq-dev
>> > >
>> > _______________________________________________
>> > zeromq-dev mailing list
>> > zeromq-dev at lists.zeromq.org
>> > http://lists.zeromq.org/mailman/listinfo/zeromq-dev
>>
>> _______________________________________________
>> zeromq-dev mailing list
>> zeromq-dev at lists.zeromq.org
>> http://lists.zeromq.org/mailman/listinfo/zeromq-dev
>>
>>
>>
>> _______________________________________________ zeromq-dev mailing list
>> zeromq-dev at lists.zeromq.org
>> http://lists.zeromq.org/mailman/listinfo/zeromq-dev
>>
>> _______________________________________________
>> zeromq-dev mailing list
>> zeromq-dev at lists.zeromq.org
>> http://lists.zeromq.org/mailman/listinfo/zeromq-dev
>>
>
>
> _______________________________________________
> zeromq-dev mailing list
> zeromq-dev at lists.zeromq.org
> http://lists.zeromq.org/mailman/listinfo/zeromq-dev
>



More information about the zeromq-dev mailing list