[zeromq-dev] app_threads in zmq_init() call

[Martin Sustrik]

Hi Jarred,

>     Thus I would just estimate how many threads are going to use 0MQ in
>     parallel (10, 100, 1000?) and set the context accordingly.
> 
> 
> You start getting invalid value when you set app_threads above 63. Line 
> 226 in zmq.cpp. I imagine there is a good reason for this limitation?


It's because the notifications from I/O thread to application thread are 
sent via lock-free polling mechanism (ypollset_t).

Lock-free algorithms are based on atomic operations and atomic 
operations work on a single word of data (32 bits on 32-bit platform, 64 
bits on 64 bit platform.

Thus, maximum number of flags that can be signaled using ypollet_t is 64 
(or 32 depending on platform). One bit is used for internal purposes so 
there are just 63 bits left to signal individual threads.

Now, there's an option to use simple socketpair for signaling between 
threads instead of ypollset_t. Doing so would add ~2us to the latency, 
however, it would allow us to move arbitrary number of bits between threads.

Actually this option is already implemented. When you init 0MQ library 
with ZMQ_POLL flag, what happens is that that ypollset_t notification 
mechanism is replaced by a socket-based notification mechanism 
(fd_signaler_t).

Currently, fd_signealer_t passes single byte between threads, meaning 
that there can be at most 256 threads. This can be changed to 2 bytes 
giving max. 65536 threads.

So, the hack to make it work for up to 256 threads is:

1. Use ZMQ_POLL flag when initialising the context.
2. Remove the check in zmq_init that complains about more than 63 threads.

As for the systematic solution I am thinking of ditching the lock-free 
polling (ypollset_t) altogether. That way we can trade 2us of extra 
latency for:

1. No need for ZMQ_POLL flag.
2. Arbitrary number of threads can be used.

Thoughts?
Martin