[zeromq-dev] HWM behaviour & blocking

Tue May 15 09:20:14 CEST 2012

Peter,

I am using mdclient2 (DEALER/ROUTER) and it is excellent, I don't have to manage IP, session,... However without ACK from the worker, the sender wouldn't know if message is received or lost during transit. I want to deploy pattern over the internet, therefore deliver confirmation is important.

My implementation requires worker to ACK for each message it received. The sender waits to receive ACK before sending the next message. I want to improve the performance by sending a batch of messages, then check ACKs, if any ACK is missed, resend missing messages.

Cheers,

Viet

Viet Hoang | Chief Executive | Quant Edge Corp

mobile: +849 8822 3399
email: viet.hoang at quant-edge.com
skype: viet.hoang.qe

Floor 1A, Viet Hai Building, Block C2H
Duy Tan Street, Cau Giay Dist
Hanoi, Vietnam

tel: +84 (4) 8587 6467
fax: +84 (4) 3795 9142
web: www.quant-edge.com

---------------------

This message (including attachments, if any) is confidential, may be privileged and is intended for the above-named recipient(s) only. If you have received this message in error, please notify me by return email and delete this message from your system. Any unauthorized use or disclosure of this message is strictly prohibited.

Quant Edge Corp assumes no responsibility for errors, inaccuracies or omissions in these materials, and does not warrant the accuracy or completeness of the information contained within these materials. Quant Edge Corp shall not be liable for any special, indirect, incidental, or consequential damages, including without limitation losses, lost revenues, or lost profits that may result from these materials.

On May14, 2012, at 11:11 PM, Pieter Hintjens wrote:

> Viet,
> 
> There is actually a variant of the MDP client that works
> asynchronously. You still want workers to be synchronous, but clients
> can stream multiple requests and get replies. See the mdpclient2
> example.
> 
> -Pieter
> 
> On Mon, May 14, 2012 at 10:36 AM, Viet Hoang (Quant Edge)
> <viet.hoang at quant-edge.com> wrote:
>> The Major Domo pattern may suits this well. Receiver to ACK for each
>> message. If you don't receive, just resend. However you sacrifice the beauty
>> of ZeroMQ - SPEED. We applied Major Domo into our demo platform and so far
>> so good (better than our old C# raw socket implementation). What we will do
>> is to have async ACK to improve performance.
>> 
>> 
>> On May 11, 2012, at 6:44 AM, Michel Pelletier wrote:
>> 
>> On Thu, May 10, 2012 at 1:53 PM, Pieter Hintjens <ph at imatix.com> wrote:
>> 
>> On Thu, May 10, 2012 at 3:44 PM, Paul Colomiets <paul at colomiets.name> wrote:
>> 
>> 
>> Can you be more specific, why setting HWM to 1 is a bad thing? Do you
>> 
>> mean, that it smells bad to set HWM to 1 for reliability? Or do you
>> 
>> think that setting it will have other consequences? (low performance?)
>> 
>> 
>> it's bad because you're trying to force a synchronous model on an
>> 
>> asynchronous system, and doing it at the wrong level. If you really
>> 
>> want synchronization you MUST get some upstream data from the
>> 
>> receiver. Just throttling the sender cannot work reliably.
>> 
>> 
>> Agreed.  Here's my take on what trips a lot of people up with 0mq:  we
>> are used to controlling how and when something is sent at the point
>> that we call "send()", or at least knowing in advance what will happen
>> if we try, but in an async model you have to let that go.  send() is
>> going to return immediately (if you haven't hit a blocking case) and
>> your message is now on its own, free as a bird, to live in various
>> queues and buffers before it ends up at its destination.  You have no
>> control or visibility of its fate after you send it unless your
>> receiver acknowledges it, or acknowledges it didn't receive it after a
>> period of time (nack).
>> 
>> The blocking case isn't really an exception, you sent when your
>> application wasn't ready to receive, either because your buffers were
>> full or your receivers weren't ready.  Senders and receivers should
>> synchronize this application level state with each other, possibly via
>> some out-of-band channel, either by indicating they are ready, or
>> connected, or that they are busy and can't do anymore, or by
>> exchanging some kind of flow control information so that the sender
>> doesn't fill the buffers because the receiver can't keep up.
>> 
>> To use an analogy, all 0mq provides are the pipes.  The pipes can't
>> tell you that the tap is running and the sink is overflowing or the
>> drain is clogged.  If you want to have a reservoir at some point to
>> regulate flow, an inline "device" can store a certain capacity of
>> messages.  If your pipe is delivering to a downstream reservoir which
>> is near full capacity, someone at the downstream end needs to pickup
>> the phone (yet another pipe of sorts) and tell the upstream to turn
>> down the flow.  If that doesn't happen, the reservoir is full, and the
>> flow stops (blocked) or maybe spills over (discards) depending on the
>> design of the *application*, not the pipe.  In either case it's not
>> the pipe's fault, it did its job, it can't solve your design problems
>> any more than a pipe can become a city water system all by itself.
>> Adding all these application level flow semantics to the "pipe" is not
>> right, and would horribly complicate the library.
>> 
>> -Michel
>> _______________________________________________
>> zeromq-dev mailing list
>> zeromq-dev at lists.zeromq.org
>> http://lists.zeromq.org/mailman/listinfo/zeromq-dev
>> 
>> 
>> 
>> _______________________________________________
>> zeromq-dev mailing list
>> zeromq-dev at lists.zeromq.org
>> http://lists.zeromq.org/mailman/listinfo/zeromq-dev
>> 
> _______________________________________________
> zeromq-dev mailing list
> zeromq-dev at lists.zeromq.org
> http://lists.zeromq.org/mailman/listinfo/zeromq-dev

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.zeromq.org/pipermail/zeromq-dev/attachments/20120515/ce0c7a8f/attachment.htm>