[zeromq-dev] "norm_engine" notes

Pieter Hintjens ph at imatix.com
Sat Mar 15 08:03:01 CET 2014


A good way to post material like this is to write it up on the wiki at
zeromq.org and then just provide a link by email.


On Sat, Mar 15, 2014 at 3:22 AM, Brian Adamson
<brian.adamson at nrl.navy.mil> wrote:
> As mentioned in the thread with the question regarding the
> "encoder::has_data()" method deprecation, here are some notes on the
> "norm_engine" adaptation of the NORM protocol for the current libzmq code.
> This a shortened overview of the protocol and our implementation.   The sort
> of "connection free" transport model and decoupling of reliability,
> congestion control, and flow control aspects that NORM provides can be
> useful for a number of different ZeroMQ messaging and other application
> goals for both multicast _and_ unicast.  The initial "norm_engine" somewhat
> replicates the capability that the PGM transport  provides albeit with
> automated congestion control and the highly efficient packet erasure FEC
> coding ARQ protocol that NORM uses along with support for unicast as well as
> multicast PUB/SUB.
> Notes for the ZeroMQ "norm_engine"
> The "norm_engine.hpp" and "norm_engine.cpp" files provide an implementation
> of a NACK-Oriented Reliable Multicast (NORM) transport protocol  option for
> ZeroMQ.  NORM is an IETF open standards protocol specified in RFC 5740 and
> supporting documents.  The Naval Research Laboratory (NRL) provides and open
> source reference implementation that is hosted at
> http://www.nrl.navy.mil/itd/ncs/products/norm.
> NORM supports reliable data delivery over IP multicast but also supports
> unicast (point to point) data transfers. NORM operates on top of the User
> Datagram Protocol (UDP) and supports reliability via NACK-based Automated
> Repeat Request (ARQ) that uses packet erasure coding for very efficient
> group communication and also provides for automated TCP-friendly congestion
> control and mechanisms for support end-to-end flow control.  NORM can also
> be configured to provide basic UDP-like best effort transport service and
> this can be enhanced by adding some amount application-settable proactive
> forward error correction (FEC) packets to the transmission.  I.e., by
> default NORM only sends 'reactive' FEC repair packets in response to NACKs
> but can also be configured to proactively send added repair packets for a
> level of reliability without any feedback from the receivers.  In addition
> its TCP-friendly congestion control, NORM can also be configured for
> fixed-rate operation and the NRL implementation supports some additional
> automated congestion control options suitable for use in bit error prone
> wireless communication environments.  While its reliable ARQ operation is
> principally NACK-based (Negative Acknowledge when packet loss is detected),
> it also supports optional positive acknowledgment (ACK) from receivers that
> can be used for delivery confirmation and flow control.
> By keeping the end-to-end transport aspects of reliability, congestion
> control, and flow control as separable components of protocol operation,
> NORM can be configured to meet a wide variety of application data delivery
> needs.  This ranges from "object-based" delivery of bulky content items like
> files and large memory segments to messaging and even real-time, reliable
> streaming.  NORM has been applied in experimental applications for file
> delivery, messaging (including serverless multicast chat and file/image
> sharing), TCP flow proxying, and real-time video and even Voice over IP
> (VoIP) streaming.
> The NRL implementation provides a procedural API compatible with C/C++ and
> also provides Java and Python (pynorm) language bindings.  The NORM code can
> be built on Linux, BSD/MacOSX, Windows, iOS, Android, and several other Unix
> operating system variants.  NORM can be built as a static or shared library
> (e.g. libnorm.so, .libnorm.dylib, norm.dlll, etc) and a "normApi.h" header
> file is provided for the C/C++ API.  The "norm_engine" addition to libzmq
> uses this API.  A comprehensive NORM Developer's reference guide and a
> growing number of simple examples is included in the source code
> distribution.  NRL also hosts a NORM developer's mailing list.
> This initial use of NORM within ZeroMQ supports the ZMQ_PUB/ZMQ_SUB and
> ZMQ_XPUB/ZMQ_XSUB socket types in a similar manner to the Pragmatic General
> Multicast (PGM) transport option that was already present in the ZeroMQ
> code.  A notable difference is NORM's support of automated TCP-friendly
> congestion control although NORM can also be configured to support
> fixed-rate (or bounded-rate) operation if desired.  Unicast addresses can
> also be used for NORM session addresses and "many-to-one" sender to receiver
> relationships can be established with a single NORM ZeroMQ subscriber
> receiving unicast messages from multiple senders.  The many additional
> transport options that NORM provides (e.g. UDP-like service, etc) could also
> be supported with extension of ZeroMQ socket options or establishment of
> multiple NORM protocol profiles.  It is hoped that the ZeroMQ community will
> be interested in exploring these options.
> The current NORM code has been tested with the Python "pyzmq" API binding.
> The basic transport endpoint format for the "norm_engine" is similar to that
> used for PGM consisting of:
> norm://[<interfaceName>;]<addr>:<port>
> where the optional <interfaceName> field provides a means to specify a
> specific host network interface for IP multicast group join and packet
> transmissions.  The <addr> can be an IP unicast or multicast address (NORM
> supports IPv4 and IPv6).  A ZeroMQ subscriber can use its loopback address
> (e.g. 127.0.0,1) for unicast sessions and multiple ZeroMQ publishers can
> "connect" to that subscriber.  In fact it is even possible for a NORM
> multicast subscriber to also terminate unicast connections for mixed-mode
> (unicast and multicast) sessions.
> One aspect of the NORM protocol is that NORM session participants must use
> unique "NormNodeId" identifiers for the NORM transport.  The default
> behavior of the NORM implementation is to auto-select a NormNodeId based on
> the host IP address.  However, if multiple NORM nodes are co-located on the
> same host, this can be problematic.  As an initial means to resolve this
> issue, the "norm_engine" transport endpoint specifier also has an optional
> "nodeId" field that can be inserted, with a comma delimiter, as the first
> field.  Thus, the full endpoint specification format is:
> norm://[<nodeId>,][<interfaceName>;]<addr>:<port>
> where both the <nodeId> and <interfaceName> are optional fields.  The
> NormNodeId is a 32-bit identifier. Its purpose is similar to the Real-time
> Protocol (RTP) synchronization source (SSRC) identifier to provide an
> address-independent means to identify the multicast source in potentially
> multi-homed or asymmetric network environments.  A future version of the
> "norm_engine" may use a hash of the ZeroMQ Identity or other means to
> establish unique NormNodeId values for multicast participants.
> The following snippets of Python "pyzmq" code illustrates  simple NORM-based
> subscription and publication (Note this includes the optional "node id"
> field so that a loopback test on a single machine will work):
> # Subscriber
> import pyzmq
> context = zmq.Context()
> socket = context.socket(zmq.SUB)
> socket.bind("norm://2,")
> socket.setsockopt(zmq.SUBSCRIBE, "ZMQ-Test")
> while True:
>     string = socket.recv()
>     print string
> # Publisher
> import pyzmq
> import time
> context = zmq.Context()
> socket = context.socket(zmq.PUB)
> socket.connect("norm://1,")
> i = 1
> while True:
>     topic = "ZMQ-Test"
>     message = "Hello, NORM " + str(i) + " ..."
>     socket.send("%s %s" % (topic, message))
>     i += 1
>     time.sleep(1)
> Note, as with the PGM transport option, the ZeroMQ socket "connect()" and
> "bind()" methods are interchangeable since NORM is a "connection-free"
> protocol.  I hate to call it "connectionless" since receivers do establish
> state on a per-sender basis, but no explicit connection setup handshake is
> required to begin data transfer.
> Regarding the "connection" establishment, NORM receivers (ZeroMQ subscribers
> in this case) "synchronize" to NORM sender (ZMQ publishers)  transmissions
> upon reception of NORM packets.  Because it was designed for IP multicast,
> "late-joining" receivers can synchronize to a NORM sender that is already in
> progress.  NORM supports multiple receiver "synchronization policies".  The
> current "norm_engine" is hard coded to use the NORM_SYNC_STREAM policy.
> This policy allows the newly joining receiver to request retransmission of
> all content the sender has buffered/cached (e.g. possibly to the beginning
> of the stream).  Another relevant NORM sync policy is NORM_SYNC_CURRENT
> where the receiver attempts reliable reception for current and newer content
> the sender is transmitting and does not request retransmission of older
> content.  Synchronization policy choice may be something that future
> versions of the ZeroMQ "norm_engine" may wish to expose.  Similarly,
> sender/receiver buffer and cache sizes (including UDP socket buffer sizes)
> are configurable along with many other details of NORM transport operation.
> The NORM_OBJECT_STREAM transport mode was selected because it enforces
> in-order delivery of transmitted content from the sender to the receiver(s).
> NORM also supports other "bulk" object data delivery models that can provide
> out-of-order message/item delivery depending on network conditions.
> The following list summarizes some of key features of NORM and this
> "norm_engine" extension to libzmq:
> NORM supports reliable multicast and unicast via a highly-efficient
> NACK-based ARQ protocol using FEC-based packet erasure coding for repair
> transmissions in response to negative acknowledgments.  NORM rides on top of
> UDP.  It can also provide UDP-like "best effort" service and this can be
> supplemented with added FEC packet erasure coding packets for something
> "better than best effort" at the expense of modest transmission overhead.
> NORM supports fixed-rate and automated, TCP-friendly congestion control
> operation.  NORM also supports some experimental congestion control options
> useful for wireless network environments.  The automated congestion control
> operation can also be rate-bounded (min/max rate excursion) if desired.
> NORM's separable reliability, congestion control, and flow control aspects
> offer support for a wide range of application data delivery needs.  This has
> included everything from bulk file delivery via IP multicast to unicast TCP
> stream proxying and even real-time quasi-reliable video and VoIP delivery.
> The current "norm_engine" assumes a fairly specific form of NORM transport
> options.  It may be desirable to expose additional NORM features and
> transport modalities via the ZeroMQ API.  Some examples include:
> Different NORM receiver sync policies and sender/receiver buffering and
> cache sizes
> Fixed-rate vs. automated congestion control options
> Specifying the NORM receiver set to the sender to allow for explicit
> positive acknowledgment (ACK) based flow control and delivery confirmation.
> While the current "norm_engine" default NORM delivery model is highly
> reliable, at more extreme bandwidth*delay, packet loss factors, the
> ACK-based flow control provides a higher guarantee of data delivery.  The
> ACK mechanism NORM uses easily scales well to multicast groups with 100's of
> nodes.
> NORM's ability to alternatively provide UDP-like "best effort" and "better
> than best effort" (using packet erasure coding) delivery services for
> applications.  This would include control of the NORM FEC encoding
> parameters.
> NORM file caching, bulk file delivery options
> Consideration of NORM use for other ZMQ socket types in addition to pub/sub.
> The current NORM API is a fairly "low level" where some of the likely useful
> "use patterns" (e.g. positive acknowledgment management) are left to the
> application programmer to implement.  The NORM API will continue to evolve
> to expose additional features as needed and to provide more convenience
> functions for implementing common "use patterns" in a simpler manner.  The
> ZeroMQ API provides a simple means to make use of NORM for different
> purposes.  As the ZeroMQ community considers and explores NORM use, the
> evolution of the NORM API and its ZeroMQ adaptation can be expanded.
> Brian Adamson
> mailto:brian.adamson at nrl.navy.mil
> 12 March 2014
> On Mar 14, 2014, at 8:11 PM, Steven McCoy <steven.mccoy at miru.hk> wrote:
> On 14 March 2014 18:43, Brian Adamson <brian.adamson at nrl.navy.mil> wrote:
>> Also, I have a short overview/write-up of what I've done with ZeroMQ/NORM
>> that I can post to the list if there's interest?
>  I'd like to see this.  I'd like to see what other features of NORM you can
> fit in and their value.
> --
> Steve-o
> _______________________________________________
> zeromq-dev mailing list
> zeromq-dev at lists.zeromq.org
> http://lists.zeromq.org/mailman/listinfo/zeromq-dev
> _______________________________________________
> zeromq-dev mailing list
> zeromq-dev at lists.zeromq.org
> http://lists.zeromq.org/mailman/listinfo/zeromq-dev

More information about the zeromq-dev mailing list