[zeromq-dev] app_threads in zmq_init() call
sustrik at 250bpm.com
Sun Apr 18 12:35:38 CEST 2010
> Thus I would just estimate how many threads are going to use 0MQ in
> parallel (10, 100, 1000?) and set the context accordingly.
> You start getting invalid value when you set app_threads above 63. Line
> 226 in zmq.cpp. I imagine there is a good reason for this limitation?
It's because the notifications from I/O thread to application thread are
sent via lock-free polling mechanism (ypollset_t).
Lock-free algorithms are based on atomic operations and atomic
operations work on a single word of data (32 bits on 32-bit platform, 64
bits on 64 bit platform.
Thus, maximum number of flags that can be signaled using ypollet_t is 64
(or 32 depending on platform). One bit is used for internal purposes so
there are just 63 bits left to signal individual threads.
Now, there's an option to use simple socketpair for signaling between
threads instead of ypollset_t. Doing so would add ~2us to the latency,
however, it would allow us to move arbitrary number of bits between threads.
Actually this option is already implemented. When you init 0MQ library
with ZMQ_POLL flag, what happens is that that ypollset_t notification
mechanism is replaced by a socket-based notification mechanism
Currently, fd_signealer_t passes single byte between threads, meaning
that there can be at most 256 threads. This can be changed to 2 bytes
giving max. 65536 threads.
So, the hack to make it work for up to 256 threads is:
1. Use ZMQ_POLL flag when initialising the context.
2. Remove the check in zmq_init that complains about more than 63 threads.
As for the systematic solution I am thinking of ditching the lock-free
polling (ypollset_t) altogether. That way we can trade 2us of extra
1. No need for ZMQ_POLL flag.
2. Arbitrary number of threads can be used.
More information about the zeromq-dev