[zeromq-dev] Thoughts on shared/multiplexing transportation channel

[Jacky Wang]

For the TCP multiplexing case, it increase the possibility of smart
batching thus may improve the system throughout.

- http://www.zeromq.org/area:faq
"How come ØMQ has higher throughput than TCP although it's built on top of TCP?"
"In other words, sending two messages down the networking stack in one
go takes much less time then sending each of them separately."

- http://mechanical-sympathy.blogspot.com/2011/10/smart-batching.html
+----------------+-----------+--------------+------------+--------------+
|                | Best (us) | Average (us) | Worst (us) | Packets Sent |
+----------------+-----------+--------------+------------+--------------+
| Serial         |       100 |          500 |      1,000 |           10 |
| Smart Batching |       100 |          150 |        200 |          1-2 |
+----------------+-----------+--------------+------------+--------------+

Thanks in advance,
Jacky

On Fri, Feb 24, 2012 at 2:05 AM, Jacky Wang <jacky.chao.wang at gmail.com> wrote:
> Hi there,
>
> I'm a new user of zmq, and I found its performance is really impressive!
>
> ./inproc_thr 1024 100000
> message size: 1024 [B]
> message count: 100000
> mean throughput: 4480086 [msg/s]
> mean throughput: 36700.865 [Mb/s]
>
> ./local_thr ipc:///tmp/labc 1024 100000
> message size: 1024 [B]
> message count: 100000
> mean throughput: 1522093 [msg/s]
> mean throughput: 12468.986 [Mb/s]
>
> ./local_thr tcp://127.0.0.1:5555 1024 100000
> message size: 1024 [B]
> message count: 100000
> mean throughput: 1136647 [msg/s]
> mean throughput: 9311.412 [Mb/s]
>
> Now I'm planning to implement a RPC system by zmq+protobuf, which has
> a brief architecture as following:
>
> +-------------+  +-------------+  +-------------+
> | Client API  |  | Client API  |  | Client API  |
> +-------------+  +-------------+  +-------------+  ......
> |     tcp     |  |     tcp     |  |     tcp     |
> +------+------+  +------+------+  +------+------+
>       #                #                #
>       #                #                #  => A) tcp multiplexing
>       #                #                #
> +-----------------------+-----------------------+
> |                      tcp                      |
> +-----------------------------------------------+
> |               Server/Dispatcher               |
> +-----------------------------------------------+
> |                    inproc                     |
> +-----------+----------------------+------------+
>            #                      #
>            #                      #  => B) inproc multiplexing
>            #                      #
>   +--------+-------+      +-------+--------+
>   |     inproc     |      |    inproc      |
>   +----------------+      +----------------+  ......
>   | Worker/Service |      | Worker/Service |
>   +----------------+      +----------------+
>
> If my understanding is right, it seems like given the destinations in
> step A or B are the same, multiple 1:1 transportation channels
> (inproc/tcp) will be created.  This could impact both performance and
> limited resource usage (especially file describers).
> - Hot RPC server may need to talk with >5K client boxes, and each of
> them may spawn >10 threads thus generate 50K connections.  A typical
> case is that client runs in Apache+PHP process.
>
> Therefore I'll be very curious to see if the transportation channel
> sharing/multiplexing is on the zmq 4.0/dev roadmap?  Some rough ideas
> include:
> - For inproc channel, as this is not a 1 provider/1 consumer any more,
> LMAX/Disruptor could be a good candidate
> (http://lists.zeromq.org/pipermail/zeromq-dev/2011-October/013673.html).
> - For tcp channel, although TCP multiplexing has its complexity
> (http://www.250bpm.com/multiplexing), we may still use an inproc-based
> queue + aggregator to solve it.
>
> Any thoughts?
>
> Thanks for bring us this awesome zmq! : )
>
> Regards,
> Jacky