[zeromq-dev] using majordomo broker with asynchronous clients
André Caron
andre.l.caron at gmail.com
Thu Dec 11 17:52:09 CET 2014
You are observing lost replies because the broker is sending replies faster
than the client is consuming them: each time the broker sends a reply, it
is stored in an internal queue for that client. When the broker sends more
replies than the client can consume, this internal queue fills up to
tolerate shorts bursts (the client's RCVHWM and the broker's SNDHWM limit
indirectly control this tolerance). However, if the broker's send rate is
constantly faster than the client's recv rate, you need infinite memory to
keep avoid losing replies.
In this scenario, the clients are operating in request-reply, so if this
happens, they are sending requests faster than they can consume the
replies. Basically, your client is sending requests until both the
client's reply queue and the broker's reply queue (for that client) are
saturated and then, instead of reading replies, it continues sending
requests. This doesn't make any sense in any real world scenario. The
"real" problem here is that the test your are performing is not realistic.
Note that it would be different if it was pub-sub instead of req-rep
because the client would not be in control of the broker's "reply" rate.
To fix this to match a real-world scenario, you should modify your client
program such that it has at most N outstanding requests, where N is
guaranteed to fit in the cilent's reply queue and broker's reply queue:
- initially send N requests;
- each time you get a reply, send a new request.
If you do this and the client's RCVHWM and broker's SNDHWM are sufficiently
high to hold a total of N replies, then you will go down to 0 lost replies
-- unless you hit other problems like disconnections, of course.
Instead of doing that, you could technically increase the server's SNDHWM
to hold 100,000 replies per client (assuming the broker has enough memory),
but the problem same problem would creep up again if you increase the
number of outstanding requests to more than 100,000.
Cheers,
André
On Thu, Dec 11, 2014 at 8:05 AM, Vishal Ahuja <vahuja4 at gmail.com> wrote:
> Sure, but when I have 3 parallel clients (300k messages), I am observing
> packet loss.
>
> Sincerely,
>
> Vishal
>
> On Thu, Dec 11, 2014 at 5:43 PM, André Caron <andre.l.caron at gmail.com>
> wrote:
>
>> Throughput is bound by multiple factors. In your case, my guess is that
>> your bottleneck is the throughput of a single TCP socket on your system. I
>> doubt that you can reach millions of messages per second with only 3
>> clients. If you add more clients, you should see the server's throughput
>> increase well over 30k.
>>
>> Cheers,
>>
>> André
>>
>> On Thu, Dec 11, 2014 at 4:40 AM, Vishal Ahuja <vahuja4 at gmail.com> wrote:
>>
>>> I removed the printfs, and increased the TCP send and receive buffers to
>>> the max possible. When I have two clients (running in parallel) sending
>>> 100k messages, there is no packet loss. But when I increase the number of
>>> clients to three, then each client receives around 93k packets. Also, the
>>> majordomo specification (http://rfc.zeromq.org/spec:7) says that the
>>> broker should be able to switch millions of messages per second, whereas I
>>> seeing a throughput of around 30k messages per second. Can someone tell how
>>> to improve the throughput?
>>>
>>> Sincerely,
>>>
>>> Vishal
>>>
>>> On Thu, Dec 11, 2014 at 10:37 AM, Andre Caron <andre.l.caron at gmail.com>
>>> wrote:
>>>
>>>> The broker uses a router socket, which silently drops outbound messages
>>>> when the sndhwm is reached. If your client is reading more slowly than the
>>>> broker is writing, you'll lose messages.
>>>>
>>>> In your case, the slow part is writing to the terminal. Try removing
>>>> the printf in the recv loop and you should lose fewer messages.
>>>>
>>>> André
>>>>
>>>> > On Dec 10, 2014, at 9:40 PM, Vishal Ahuja <vahuja4 at gmail.com> wrote:
>>>> >
>>>> > While reading the zeromq guide, I came across client code (pertaining
>>>> to the majordomo pattern) which sends 100k requests in a loop, and then
>>>> receives the reply in a second loop.
>>>> >
>>>> > #include "../include/mdp.h"
>>>> > #include <time.h>
>>>> >
>>>> >
>>>> > int main (int argc, char *argv [])
>>>> > {
>>>> > int verbose = (argc > 1 && streq (argv [1], "-v"));
>>>> > mdp_client_t *session = mdp_client_new
>>>> ("tcp://localhost:5555", verbose);
>>>> > int count;
>>>> > for (count = 0; count < 100000; count++) {
>>>> > zmsg_t *request = zmsg_new ();
>>>> > zmsg_pushstr (request, "Hello world");
>>>> > mdp_client_send (session, "echo", &request);
>>>> > }
>>>> > printf("sent all\n");
>>>> >
>>>> > for (count = 0; count < 100000; count++) {
>>>> > zmsg_t *reply = mdp_client_recv (session,NULL,NULL);
>>>> > if (reply)
>>>> > zmsg_destroy (&reply);
>>>> > else
>>>> > break; // Interrupted by Ctrl-C
>>>> > printf("reply received:%d\n", count);
>>>> > }
>>>> > printf ("%d replies received\n", count);
>>>> > mdp_client_destroy (&session);
>>>> > return 0;
>>>> > }
>>>> >
>>>> > I have added a counter to count the number of replies that the worker
>>>> (test_worker.c) sends to the broker, and another counter in mdp_broker.c to
>>>> count the number of replies the broker sends to a client. Both of these
>>>> count up to 100k, but the client is receiving only around 37k replies.
>>>> >
>>>> > If the number of client requests is set to around 40k, then it
>>>> receives all the replies. Can someone please tell me why packets are lost
>>>> when the client sends more than 40k asynchronous requests?
>>>> >
>>>> > I tried setting the HWM to 100k for the broker socket, but the
>>>> problem persists:
>>>> >
>>>> > static broker_t *
>>>> > s_broker_new (int verbose)
>>>> > {
>>>> > broker_t *self = (broker_t *) zmalloc (sizeof (broker_t));
>>>> > int64_t hwm = 100000;
>>>> > // Initialize broker state
>>>> > self->ctx = zctx_new ();
>>>> > self->socket = zsocket_new (self->ctx, ZMQ_ROUTER);
>>>> > zmq_setsockopt(self->socket, ZMQ_SNDHWM, &hwm, sizeof(hwm));
>>>> >
>>>> > zmq_setsockopt(self->socket, ZMQ_RCVHWM, &hwm, sizeof(hwm));
>>>> > self->verbose = verbose;
>>>> > self->services = zhash_new ();
>>>> > self->workers = zhash_new ();
>>>> > self->waiting = zlist_new ();
>>>> > self->heartbeat_at = zclock_time () + HEARTBEAT_INTERVAL;
>>>> > return self;
>>>> > }
>>>> >
>>>> > Vishal
>>>> > _______________________________________________
>>>> > zeromq-dev mailing list
>>>> > zeromq-dev at lists.zeromq.org
>>>> > http://lists.zeromq.org/mailman/listinfo/zeromq-dev
>>>> _______________________________________________
>>>> zeromq-dev mailing list
>>>> zeromq-dev at lists.zeromq.org
>>>> http://lists.zeromq.org/mailman/listinfo/zeromq-dev
>>>>
>>>
>>>
>>> _______________________________________________
>>> zeromq-dev mailing list
>>> zeromq-dev at lists.zeromq.org
>>> http://lists.zeromq.org/mailman/listinfo/zeromq-dev
>>>
>>>
>>
>> _______________________________________________
>> zeromq-dev mailing list
>> zeromq-dev at lists.zeromq.org
>> http://lists.zeromq.org/mailman/listinfo/zeromq-dev
>>
>>
>
> _______________________________________________
> zeromq-dev mailing list
> zeromq-dev at lists.zeromq.org
> http://lists.zeromq.org/mailman/listinfo/zeromq-dev
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.zeromq.org/pipermail/zeromq-dev/attachments/20141211/e27d8a4c/attachment.htm>
More information about the zeromq-dev
mailing list