[zeromq-dev] Thoughts on shared/multiplexing transportation channel

[Jacky Wang]

Hi there,

I'm a new user of zmq, and I found its performance is really impressive!

./inproc_thr 1024 100000
message size: 1024 [B]
message count: 100000
mean throughput: 4480086 [msg/s]
mean throughput: 36700.865 [Mb/s]

./local_thr ipc:///tmp/labc 1024 100000
message size: 1024 [B]
message count: 100000
mean throughput: 1522093 [msg/s]
mean throughput: 12468.986 [Mb/s]

./local_thr tcp://127.0.0.1:5555 1024 100000
message size: 1024 [B]
message count: 100000
mean throughput: 1136647 [msg/s]
mean throughput: 9311.412 [Mb/s]

Now I'm planning to implement a RPC system by zmq+protobuf, which has
a brief architecture as following:

+-------------+  +-------------+  +-------------+
| Client API  |  | Client API  |  | Client API  |
+-------------+  +-------------+  +-------------+  ......
|     tcp     |  |     tcp     |  |     tcp     |
+------+------+  +------+------+  +------+------+
       #                #                #
       #                #                #  => A) tcp multiplexing
       #                #                #
+-----------------------+-----------------------+
|                      tcp                      |
+-----------------------------------------------+
|               Server/Dispatcher               |
+-----------------------------------------------+
|                    inproc                     |
+-----------+----------------------+------------+
            #                      #
            #                      #  => B) inproc multiplexing
            #                      #
   +--------+-------+      +-------+--------+
   |     inproc     |      |    inproc      |
   +----------------+      +----------------+  ......
   | Worker/Service |      | Worker/Service |
   +----------------+      +----------------+

If my understanding is right, it seems like given the destinations in
step A or B are the same, multiple 1:1 transportation channels
(inproc/tcp) will be created.  This could impact both performance and
limited resource usage (especially file describers).
- Hot RPC server may need to talk with >5K client boxes, and each of
them may spawn >10 threads thus generate 50K connections.  A typical
case is that client runs in Apache+PHP process.

Therefore I'll be very curious to see if the transportation channel
sharing/multiplexing is on the zmq 4.0/dev roadmap?  Some rough ideas
include:
- For inproc channel, as this is not a 1 provider/1 consumer any more,
LMAX/Disruptor could be a good candidate
(http://lists.zeromq.org/pipermail/zeromq-dev/2011-October/013673.html).
- For tcp channel, although TCP multiplexing has its complexity
(http://www.250bpm.com/multiplexing), we may still use an inproc-based
queue + aggregator to solve it.

Any thoughts?

Thanks for bring us this awesome zmq! : )

Regards,
Jacky