[zeromq-dev] inproc queue performance

[Jean-François Smigielski]

Hello Erik and folks,

Have you tried spawning multiple acceptor threads ?

I am a ZeroMQ newbie and I might be wrong but..

   - with only one acceptor, with need an intermediate device? you could
   connect workers sockets to the socket bond in the acceptor.
   - have you tried replacing the REQ/REP pattern by a PUSH/PULL pattern?
   Do you really need the pointer to be returned to the acceptor thread? I
   feel you could parallelize better your jobs in the workers, without waiting
   for a reply for each connection. What I read in your exemple is that you
   are benchmark accept() more than ZMQ.


--
Jean-François SMIGIELSKI
+33 (0) 625 135 563


2013/2/2 Erik Fears <strtok at strtok.net>

> I'm using zeromq 3.2.2, inproc sockets, and the ZMQ_QUEUE device to
> dispatch incoming
> TCP connections from an acceptor thread to 8 worker threads.
>
> I'm finding that I can only send about 8500 messages/second over the queue
> to worker threads.
>
> I've tried reducing the amount of work done outside zmq as much as
> possible to isolate the performance
> problem.
>
> Any ideas? Is the queue code expected to be this slow?
>
> Here's the setup:
>
> 1. There's a pthread that starts that has the purpose of being the queue.
> Here's how it's started:
>
>     void *frontend_socket = zmq_socket(zmq_context(), ZMQ_ROUTER);
>     void *backend_socket = zmq_socket(zmq_context(), ZMQ_DEALER);
>     zmq_bind(frontend_socket, "inproc://listener-queue-frontend");
>     zmq_bind(backend_socket,  "inproc://listener-queue-backend");
>     boost::thread dispatch_thread(boost::bind(&dispatch_thread_main,
> frontend_socket, backend_socket));
>
>
> void
> dispatch_thread_main(void *frontend_socket, void *backend_socket)
> {
>     zmq_device(ZMQ_QUEUE, frontend_socket, backend_socket);
>     zmq_close(frontend_socket);
>     zmq_close(backend_socket);
> }
>
>
> 2. There's a second thread that's purpose is to receive new connections
> (accept()), and then
> send them to the queue. Here's what the code looks like:
>
>     void *dispatch_socket = zmq_socket(zmq_context(), ZMQ_REQ);
>
>     if (zmq_connect(dispatch_socket, "inproc://listener-queue-frontend")
> != 0) {
>             // handle error
>     }
>
>     while (int new_client_fd = accept(listen_fd, (sockaddr*) &client_addr,
>                                      &client_addr_len))
>     {
>
>         //Sends to dispatcher and should immediately unblock
>         zmq_send_ptr<int>(dispatch_socket, &new_client_fd);
>
>         //Block zmq_recv_ptr will block until the reply
>         int *fd = zmq_recv_ptr<int>(dispatch_socket);
>         assert(fd == &new_client_fd);
>     }
>
>
> 3. N number of works (e.g. 8) are started that execute the following code.
> All I'm doing right now
>     is calling close() on the accepted socket in the worker thread.
>
>       void *dispatch_socket = zmq_socket(zmq_context(), ZMQ_REP);
>
>         if (zmq_connect(dispatch_socket,
> "inproc://listener-queue-backend") != 0) {
>           //handle error
>         }
>
>         int *client_fd = NULL;
>        while ((client_fd = zmq_recv_ptr<int>(dispatch_socket, 0)) !=
> nullptr) {
>                 close(*client_fd);
>                 zmq_send_ptr<int>(dispatch_socket.get(), client_fd);
>         }
>
>
> 4. zmq_recv_ptr/send_ptr are just some helper functions designed to
> implement 0copy message
>    passing of pointers. They use zmq_msg_recv/send, and zmq_msg_init_data.
> I don't think these
>    are the source of any performance issues.
>
>
>
> _______________________________________________
> zeromq-dev mailing list
> zeromq-dev at lists.zeromq.org
> http://lists.zeromq.org/mailman/listinfo/zeromq-dev
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.zeromq.org/pipermail/zeromq-dev/attachments/20130202/6ade75f3/attachment.htm>