[zeromq-dev] using majordomo broker with asynchronous clients

Vishal Ahuja vahuja4 at gmail.com
Fri Dec 12 07:57:23 CET 2014


Andre, thank you for your help! I changed the client to send 10k requests
at once, and then send two requests for every reply that it receives. I
chose 10k because that allowed me to run up to ten clients (each sending
100k messages) in parallel without any packet loss. Here is the client code:

#include "../include/mdp.h"

#include <time.h>

int main (int argc, char *argv [])

{

    int verbose = (argc > 1 && streq (argv [1], "-v"));

    mdp_client_t *session = mdp_client_new (argv[1], verbose);

    //mdp_client_t *session = mdp_client_new("tcp://localhost:5555",
verbose);

    int count1, count2;

    struct timeval start,end;

    gettimeofday(&start, NULL);

    for (count1 = 0; count1 < 10000; count1++) {

        zmsg_t *request = zmsg_new ();

        zmsg_pushstr (request, "Hello world");

        mdp_client_send (session, "echo", &request);

    }

    printf("sent all\n");

    for (count1 = 0; count1 < 45000; count1++) {

        zmsg_t *reply = mdp_client_recv (session,NULL,NULL);

        if (reply)

        {

            zmsg_destroy (&reply);

            zmsg_t *request = zmsg_new ();

            zmsg_pushstr (request, "Hello world");

            mdp_client_send (session, "echo", &request);


            request = zmsg_new ();

            zmsg_pushstr (request, "Hello world");

            mdp_client_send (session, "echo", &request);

        }

         else

            break;              //  Interrupted by Ctrl-C

    }

   /* receiving the remaining 55k replies */

    for(count1 = 45000; count1 < 100000; count1++)

    {

        zmsg_t *reply = mdp_client_recv (session,NULL,NULL);

        if (reply)

        {

            zmsg_destroy (&reply);

        }

         else

            break;


    }

    gettimeofday(&end, NULL);

    long elapsed = (end.tv_sec - start.tv_sec) +((end.tv_usec -
start.tv_usec)/1000000);

    printf("time = %ld\n", elapsed);

    printf ("%d replies received\n", count1);

    mdp_client_destroy (&session);

    return 0;

}


I ran the broker, worker, and the clients within the same machine. Here is
the recorded time:

# of clients in parallel

(each client sends 100k )                   Time elapsed (seconds)

1                                                                     4


2                                                                     9

3                                                                    12

4                                                                    16

5                                                                    21

10                                                                  43


So for every 100k requests, the broker is taking about 4 seconds. Is this
the expected behavior? Am not sure how to achieve million messages per
second. Please advise.

Sincerely,

Vishal




On Thu, Dec 11, 2014 at 10:22 PM, André Caron <andre.l.caron at gmail.com>
wrote:
>
> You are observing lost replies because the broker is sending replies
> faster than the client is consuming them: each time the broker sends a
> reply, it is stored in an internal queue for that client.  When the broker
> sends more replies than the client can consume, this internal queue fills
> up to tolerate shorts bursts (the client's RCVHWM and the broker's SNDHWM
> limit indirectly control this tolerance).  However, if the broker's send
> rate is constantly faster than the client's recv rate, you need infinite
> memory to keep avoid losing replies.
>
> In this scenario, the clients are operating in request-reply, so if this
> happens, they are sending requests faster than they can consume the
> replies.  Basically, your client is sending requests until both the
> client's reply queue and the broker's reply queue (for that client) are
> saturated and then, instead of reading replies, it continues sending
> requests.  This doesn't make any sense in any real world scenario.  The
> "real" problem here is that the test your are performing is not realistic.
>
> Note that it would be different if it was pub-sub instead of req-rep
> because the client would not be in control of the broker's "reply" rate.
>
> To fix this to match a real-world scenario, you should modify your client
> program such that it has at most N outstanding requests, where N is
> guaranteed to fit in the cilent's reply queue and broker's reply queue:
> - initially send N requests;
> - each time you get a reply, send a new request.
>
> If you do this and the client's RCVHWM and broker's SNDHWM are
> sufficiently high to hold a total of N replies, then you will go down to 0
> lost replies -- unless you hit other problems like disconnections, of
> course.
>
> Instead of doing that, you could technically increase the server's SNDHWM
> to hold 100,000 replies per client (assuming the broker has enough memory),
> but the problem same problem would creep up again if you increase the
> number of outstanding requests to more than 100,000.
>
> Cheers,
>
> André
>
> On Thu, Dec 11, 2014 at 8:05 AM, Vishal Ahuja <vahuja4 at gmail.com> wrote:
>
>> Sure, but when I have 3 parallel clients (300k messages), I am observing
>> packet loss.
>>
>> Sincerely,
>>
>> Vishal
>>
>> On Thu, Dec 11, 2014 at 5:43 PM, André Caron <andre.l.caron at gmail.com>
>> wrote:
>>
>>> Throughput is bound by multiple factors.  In your case, my guess is that
>>> your bottleneck is the throughput of a single TCP socket on your system.  I
>>> doubt that you can reach millions of messages per second with only 3
>>> clients.  If you add more clients, you should see the server's throughput
>>> increase well over 30k.
>>>
>>> Cheers,
>>>
>>> André
>>>
>>> On Thu, Dec 11, 2014 at 4:40 AM, Vishal Ahuja <vahuja4 at gmail.com> wrote:
>>>
>>>> I removed the printfs, and increased the TCP send and receive buffers
>>>> to the max possible. When I have two clients (running in parallel) sending
>>>> 100k messages, there is no packet loss. But when I increase the number of
>>>> clients to three, then each client receives around 93k packets. Also, the
>>>> majordomo specification (http://rfc.zeromq.org/spec:7) says that the
>>>> broker should be able to switch millions of messages per second, whereas I
>>>> seeing a throughput of around 30k messages per second. Can someone tell how
>>>> to improve the throughput?
>>>>
>>>> Sincerely,
>>>>
>>>> Vishal
>>>>
>>>> On Thu, Dec 11, 2014 at 10:37 AM, Andre Caron <andre.l.caron at gmail.com>
>>>> wrote:
>>>>
>>>>> The broker uses a router socket, which silently drops outbound
>>>>> messages when the sndhwm is reached.  If your client is reading more slowly
>>>>> than the broker is writing, you'll lose messages.
>>>>>
>>>>> In your case, the slow part is writing to the terminal.  Try removing
>>>>> the printf in the recv loop and you should lose fewer messages.
>>>>>
>>>>> André
>>>>>
>>>>> > On Dec 10, 2014, at 9:40 PM, Vishal Ahuja <vahuja4 at gmail.com> wrote:
>>>>> >
>>>>> > While reading the zeromq guide, I came across client code
>>>>> (pertaining to the majordomo pattern) which sends 100k requests in a loop,
>>>>> and then receives the reply in a second loop.
>>>>> >
>>>>> >     #include "../include/mdp.h"
>>>>> >     #include <time.h>
>>>>> >
>>>>> >
>>>>> >     int main (int argc, char *argv [])
>>>>> >     {
>>>>> >         int verbose = (argc > 1 && streq (argv [1], "-v"));
>>>>> >         mdp_client_t *session = mdp_client_new
>>>>> ("tcp://localhost:5555", verbose);
>>>>> >         int count;
>>>>> >         for (count = 0; count < 100000; count++) {
>>>>> >             zmsg_t *request = zmsg_new ();
>>>>> >             zmsg_pushstr (request, "Hello world");
>>>>> >             mdp_client_send (session, "echo", &request);
>>>>> >         }
>>>>> >         printf("sent all\n");
>>>>> >
>>>>> >         for (count = 0; count < 100000; count++) {
>>>>> >             zmsg_t *reply = mdp_client_recv (session,NULL,NULL);
>>>>> >             if (reply)
>>>>> >                 zmsg_destroy (&reply);
>>>>> >             else
>>>>> >                 break;              //  Interrupted by Ctrl-C
>>>>> >             printf("reply received:%d\n", count);
>>>>> >         }
>>>>> >         printf ("%d replies received\n", count);
>>>>> >         mdp_client_destroy (&session);
>>>>> >         return 0;
>>>>> >     }
>>>>> >
>>>>> > I have added a counter to count the number of replies that the
>>>>> worker (test_worker.c) sends to the broker, and another counter in
>>>>> mdp_broker.c to count the number of replies the broker sends to a client.
>>>>> Both of these count up to 100k, but the client is receiving only around 37k
>>>>> replies.
>>>>> >
>>>>> > If the number of client requests is set to around 40k, then it
>>>>> receives all the replies. Can someone please tell me why packets are lost
>>>>> when the client sends more than 40k asynchronous requests?
>>>>> >
>>>>> > I tried setting the HWM to 100k for the broker socket, but the
>>>>> problem persists:
>>>>> >
>>>>> >     static broker_t *
>>>>> >     s_broker_new (int verbose)
>>>>> >     {
>>>>> >         broker_t *self = (broker_t *) zmalloc (sizeof (broker_t));
>>>>> >         int64_t hwm = 100000;
>>>>> >         //  Initialize broker state
>>>>> >         self->ctx = zctx_new ();
>>>>> >         self->socket = zsocket_new (self->ctx, ZMQ_ROUTER);
>>>>> >         zmq_setsockopt(self->socket, ZMQ_SNDHWM, &hwm, sizeof(hwm));
>>>>> >
>>>>> >         zmq_setsockopt(self->socket, ZMQ_RCVHWM, &hwm, sizeof(hwm));
>>>>> >         self->verbose = verbose;
>>>>> >         self->services = zhash_new ();
>>>>> >         self->workers = zhash_new ();
>>>>> >         self->waiting = zlist_new ();
>>>>> >         self->heartbeat_at = zclock_time () + HEARTBEAT_INTERVAL;
>>>>> >         return self;
>>>>> > }
>>>>> >
>>>>> > Vishal
>>>>> > _______________________________________________
>>>>> > zeromq-dev mailing list
>>>>> > zeromq-dev at lists.zeromq.org
>>>>> > http://lists.zeromq.org/mailman/listinfo/zeromq-dev
>>>>> _______________________________________________
>>>>> zeromq-dev mailing list
>>>>> zeromq-dev at lists.zeromq.org
>>>>> http://lists.zeromq.org/mailman/listinfo/zeromq-dev
>>>>>
>>>>
>>>>
>>>> _______________________________________________
>>>> zeromq-dev mailing list
>>>> zeromq-dev at lists.zeromq.org
>>>> http://lists.zeromq.org/mailman/listinfo/zeromq-dev
>>>>
>>>>
>>>
>>> _______________________________________________
>>> zeromq-dev mailing list
>>> zeromq-dev at lists.zeromq.org
>>> http://lists.zeromq.org/mailman/listinfo/zeromq-dev
>>>
>>>
>>
>> _______________________________________________
>> zeromq-dev mailing list
>> zeromq-dev at lists.zeromq.org
>> http://lists.zeromq.org/mailman/listinfo/zeromq-dev
>>
>>
>
> _______________________________________________
> zeromq-dev mailing list
> zeromq-dev at lists.zeromq.org
> http://lists.zeromq.org/mailman/listinfo/zeromq-dev
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.zeromq.org/pipermail/zeromq-dev/attachments/20141212/fb04022c/attachment.htm>


More information about the zeromq-dev mailing list