[zeromq-dev] Migrating IPC-based Pub/Sub System To ZMQ

[Matt Weinstein]

I converted this to PDF YMMV:


On Aug 13, 2010, at 11:57 AM, Santy, Michael wrote:

> All,
>
> I'm currently working to refactor an existing pub/sub distributed  
> system built around CMU IPC[1] and raw sockets to solely use ZMQ.   
> I've not been able to fit it into any one of the common patterns, so  
> I'd like to get some feedback from the list on how to improve my  
> proposed design.
>
> For those of you not familiar with CMU IPC: It is a brokered pub/sub  
> framework in which every process can pub and sub through the CMU IPC  
> API.  CMU IPC connects to a "central" server process via TCP/IP to  
> register for subscriptions and publish messages.  Typically this  
> central server also routes the messages as well.  While CMU IPC  
> works relatively well for its intended use, we've found its  
> performance and fault tolerance a bit lacking.  We've done some  
> benchmarking and on large messages the performance is unacceptable.   
> In addition to the multi-hop latency forced by a brokered system,  
> the network utilization was less than 30% on 1GbE and much worse on  
> 20Gb IB.  In addition, the IPC central server is a central point of  
> failure.  If it goes down, it takes all of the clients with it and  
> we've found no way to recover.
>
> The top half of the attached diagram illustrates our existing  
> system.  It collects high-speed data from a number of sources,  
> combines the data centrally into processing packets, and farms out  
> these processing packets (mostly) round-robin to one of many data  
> processors.  Because of the performance issues we only use CMU IPC  
> for administrative messages (shown in dashed red).  We've  
> implemented a mechanism based on raw sockets for distributing the  
> high-speed data (shown in blue), and use CMU IPC to coordinate these  
> socket connections.
>
> I believe that by adopting ZMQ, we can improve / simplify our system  
> by moving both our high-speed data and administrative messages into  
> a single ZMQ-based messaging framework.  As a benefit, it appears  
> that we can gain some fault tolerance, as ZMQ abstracts the socket  
> bind/connect ordering and handles reconnection.  The lower half of  
> the diagram shows my best guess of how to adopt ZMQ.  In this  
> diagram, all connections between processes are ZMQ, and the  
> endpoints are labeled as to whether the ZMQ socket binds or  
> connects, and whether it is a pub or a sub ZMQ socket.
>
> The zmq_forwarder device seems very analogous to the CMU IPC central  
> server, so it appears easiest to have it bind for subscribers and  
> for publishers.  Each process that needs to publish or subscribe  
> would connect to one or both of the endpoints provided by  
> zmq_forwarder.  All administrative messaging will go through this  
> zmq_forwarder process.  For high-speed data, I don't think it would  
> be best to go through the forwarder, as the data will only be sent  
> to one process, and I'd like to avoid the overhead of two network  
> hops.  In this case I think it makes sense to use pub/sub ZMQ  
> sockets directly from the publisher of the high-speed data to the  
> subscriber.
>
> In this design, there are still two central points of failure: the  
> zmq_forwarder and the data_combiner.  If either of these processes  
> go down, the operation of the system will stop.  In the previous  
> system, it would stop catastrophically.  In the ZMQ-based refactored  
> system, I believe that we can gracefully handle unavailability of  
> either of these processes and restart when they come back only due  
> to ZMQ's connection abstraction.
>
> I've played around with the examples and have a superficial  
> understanding of ZMQ, but have yet to apply it to a non-academic  
> problem.  I'd really appreciate any feedback on this design from  
> those of you using ZMQ in production systems.
>
> Thanks!
>
> Mike Santy
>
> [1] http://www.cs.cmu.edu/~IPC/
>
>
>
> <zmq_topology.svgz>_______________________________________________
> zeromq-dev mailing list
> zeromq-dev at lists.zeromq.org
> http://lists.zeromq.org/mailman/listinfo/zeromq-dev

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.zeromq.org/pipermail/zeromq-dev/attachments/20100813/535381f2/attachment.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: zmq_topology.pdf
Type: application/pdf
Size: 21636 bytes
Desc: not available
URL: <https://lists.zeromq.org/pipermail/zeromq-dev/attachments/20100813/535381f2/attachment.pdf>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.zeromq.org/pipermail/zeromq-dev/attachments/20100813/535381f2/attachment-0001.htm>