[zeromq-dev] Handling high socket or connection counts

[Goswin von Brederlow]

On Thu, Feb 06, 2014 at 01:38:48PM +0100, Olaf Mandel wrote:
> Hello,
> 
> I was wondering if there is any pattern to handle many open connections
> to a single socket or many open sockets in one executable: I
> consistently run into a "Too many open files" problem.
> 
> I get this error either when I run into the ZMQ_MAX_SOCKETS limit or
> into the operation system resource limit (ulimit -n) on Linux. On
> Windows, I can be lucky not to cause a Bluescreen!
> 
> I therefore assume this is a general design question, not a specific
> problem with 0MQ.
> 
> Background: I was considering a program design consisting of many
> independent parts that run in many different threads and must exchange
> data. I was planning to expose the readable/manipulable properties via
> 0MQ REQ-REP and PUB-SUB sockets, via the inproc or tcp transport. But in
> the naive approach the number of possible sockets and connections in one
> executable is too low:
> 
> * one file descriptor per socket
> * for tcp, one file descriptor per connection
> 
> The number needed in my case ranges from a few hundred sockets to many
> thousands (configuration dependent) and I don't even know how many
> connections I might end up having...
> 
> 
> My REQ-REP test-case, stripped of all checks:
> 
> #include <vector>
> #include <zmq.h>
> int main() {
>     void* context = zmq_ctx_new();
>     void* server = zmq_socket(context, ZMQ_REP);
>     zmq_bind(server, "tcp://127.0.0.1:5555");
>     std::vector<void*> clients;
>     for(unsigned int i=0; i<2048; ++i) {
>         clients.push_back(zmq_socket(context, ZMQ_REQ));
>         zmq_connect(clients.back(), "tcp://127.0.0.1:5555");
>     }
>     return 0;
> }
> 
> The full test case is available under http://pastebin.com/hN4PYkX3 .
> 
> 
> Any insight is much appreciated!
> 
> Thank you,
> Olaf Mandel

ZMQ internally has to use the kernels socket() and connect() calls to
establish a tcp connection. This will use up file descriptors and they
are strictly limited, usualy to 1024 per program by default. So that
is a rather limited resource in a single programm. When you use
threads then you should be using inproc. That only uses a bit of
memory (right?) and no file descriptors at all. Problem solved.

But what if you don't use threads? Then inproc won't work. But also
you don't have a limit of 1024 sockets. Only 1024 file descriptors per
executable. By forking or starting seperate executables you can have
many more sockets, colletively. How much more? There are only 65536
ports and 0-1023 is reserved for root. So you won't get verry far with
tcp. If you need more then you need to use multiple hosts or at least
containers with different IPs.

Or use unix domain sockets so you are not limited by the number of
ports for tcp.

MfG
	Goswin