[zeromq-dev] inproc queue performance

[Erik Fears]

I'm using zeromq 3.2.2, inproc sockets, and the ZMQ_QUEUE device to
dispatch incoming
TCP connections from an acceptor thread to 8 worker threads.

I'm finding that I can only send about 8500 messages/second over the queue
to worker threads.

I've tried reducing the amount of work done outside zmq as much as possible
to isolate the performance
problem.

Any ideas? Is the queue code expected to be this slow?

Here's the setup:

1. There's a pthread that starts that has the purpose of being the queue.
Here's how it's started:

    void *frontend_socket = zmq_socket(zmq_context(), ZMQ_ROUTER);
    void *backend_socket = zmq_socket(zmq_context(), ZMQ_DEALER);
    zmq_bind(frontend_socket, "inproc://listener-queue-frontend");
    zmq_bind(backend_socket,  "inproc://listener-queue-backend");
    boost::thread dispatch_thread(boost::bind(&dispatch_thread_main,
frontend_socket, backend_socket));


void
dispatch_thread_main(void *frontend_socket, void *backend_socket)
{
    zmq_device(ZMQ_QUEUE, frontend_socket, backend_socket);
    zmq_close(frontend_socket);
    zmq_close(backend_socket);
}


2. There's a second thread that's purpose is to receive new connections
(accept()), and then
send them to the queue. Here's what the code looks like:

    void *dispatch_socket = zmq_socket(zmq_context(), ZMQ_REQ);

    if (zmq_connect(dispatch_socket, "inproc://listener-queue-frontend") !=
0) {
            // handle error
    }

    while (int new_client_fd = accept(listen_fd, (sockaddr*) &client_addr,
                                     &client_addr_len))
    {

        //Sends to dispatcher and should immediately unblock
        zmq_send_ptr<int>(dispatch_socket, &new_client_fd);

        //Block zmq_recv_ptr will block until the reply
        int *fd = zmq_recv_ptr<int>(dispatch_socket);
        assert(fd == &new_client_fd);
    }


3. N number of works (e.g. 8) are started that execute the following code.
All I'm doing right now
    is calling close() on the accepted socket in the worker thread.

      void *dispatch_socket = zmq_socket(zmq_context(), ZMQ_REP);

        if (zmq_connect(dispatch_socket, "inproc://listener-queue-backend")
!= 0) {
          //handle error
        }

        int *client_fd = NULL;
       while ((client_fd = zmq_recv_ptr<int>(dispatch_socket, 0)) !=
nullptr) {
                close(*client_fd);
                zmq_send_ptr<int>(dispatch_socket.get(), client_fd);
        }


4. zmq_recv_ptr/send_ptr are just some helper functions designed to
implement 0copy message
   passing of pointers. They use zmq_msg_recv/send, and zmq_msg_init_data.
I don't think these
   are the source of any performance issues.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.zeromq.org/pipermail/zeromq-dev/attachments/20130201/0d0ff77d/attachment.htm>