[zeromq-dev] Assertion failed: fds.size() <= FD_SETSIZE in select.cpp:67

[Martin Sustrik]

Hi Troy,

> I ran into this same issue on my Windows box install today. I was in
> the process of putting together a new system for handling queries on a
> web site. I had a web front end (linux), a queue device broker
> (windows), and numerous workers (windows) in the backend processing
> queries.
>
> I am using pyzmq in python for all three pieces, and using the queue
> device for the broker. I'm using 0mq version 2.0.10.
>
> During initial testing the queue/broker was crashing with the same
> 'FD_SETSIZE; error described here. This was happening after hitting it
> with only minor load (a few hundred requests in a tight loop in a
> testing app). After reading this thread I immediately assumed it was
> the Windows environment causing my issue. Our backend is Windows
> (because some of the queries require windows specific stuff). I tried
> moving the broker to one of the Linux boxes where the web front end
> lives.
>
> I encountered a similar issue ('Too many open files sock != -1
> (tcp_listener.cpp:316)' ).
>
> Then I looked in my worker code... There was a bug there that was
> responsible for this. There's a primary loop where it connects to the
> broker, grabs the message, processes, sends results, rinse repeat.
> Unfortunately the call to connect the socket was INSIDE the loop
> instead of outside where it should have been.
>
> I moved the broker back to Windows after having fixed the worker
> connection code, and the problem disappeared. I hit it with
> increasingly more load, from 100 to 100,000 queries in a tight loop
> from a single client and did not encounter any problems with the
> broker. I tried 10,000 requests each from two concurrent client
> connections, also no problem. I imagine I won't see this problem
> again, unless I open my broker to a lot more endpoints.
>
> So, I suggest, before modifying the FD_SETSIZE constant, that you
> inspect your client/worker code for correct use and disposal of socket
> connections. This may or may not help in your case.

Right. Also note that in 2.1 the FD_SETSIZE default is rised from 64 to 
1024.

Martin