[zeromq-dev] Yahoo shows big performance improvement by switching Storm from ZeroMQ to Netty

[Asquith, Edward]

Hi all,

Most of you are probably aware of Storm, which is a distributed stream-processing framework. It's built on ZeroMQ, and provides features for resilience against message loss and APIs for plugging in sources, sinks and analytic processing.

I think it's a fairly classic use case for ZeroMQ, but this interesting article from Yahoo talks about how they have obtained performance improvements by switching out ZeroMQ for Netty. After a bit of tuning, they succeeded in pretty much doubling throughput.

http://yahooeng.tumblr.com/post/64758709722/making-storm-fly-with-netty

Obviously Netty supports a narrower set of use cases, but being pure Java this certainly caught my attention; better performance and a simplified deployment model is a compelling combo.

I will  be trying to reproduce this result for my own ZMQ-based system, but I'm interested to hear what other people on this list think?

Edward/
BAE Systems Applied Intelligence
Please consider the environment before printing this email. This message should be regarded as confidential. If you have received this email in error please notify the sender and destroy it immediately. Statements of intent shall only become binding when confirmed in hard copy by an authorised signatory. The contents of this email may relate to dealings with other companies under the control of BAE Systems Applied Intelligence Limited, details of which can be found at http://www.baesystems.com/Businesses/index.htm.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.zeromq.org/pipermail/zeromq-dev/attachments/20140407/59bbdd1c/attachment.htm>