[zeromq-dev] source material for TIBCO to ØMQ migration

Steven McCoy steven.mccoy at miru.hk
Tue Jan 4 04:31:10 CET 2011


On 4 January 2011 10:40, Scott <alcoholiday at gmail.com> wrote:

> I obviously need to read up on PGM. I'd imagine some combination of
> forward EC + no back channel would probably satisfy our applications,
> I'd probably be pretty happy w/ a 50% FEC penalty in exchange for no
> back channel.
>
>
FEC is pretty amazing tech but for OpenPGM I don't offer commercial support
until a client comes forward with a reasonable requirement to complete
production quality.  The reason being is that I have implemented one FEC
protocol on reading the draft specification and the upstream professors
notes on software FEC (Luigi Rizzo) but Microsoft & Whitebarn implemented a
separate reading of the specification even though Microsoft uses Rizzo's
code.  The latter allows for stream support but is incompatible with non-FEC
receivers and provides a false sense of reliability as you have to push very
large messages in order to actually get FEC running.  The OpenPGM
implementation is compatible with non-FEC receivers, always uses FEC, but
requires awareness of transmission-group sizes in order to not stall
recovery.

Performance is another issue, Microsoft & Whitebarn PGM really have issues
even running non-FEC at 10,000 packets-per-second, adding FEC drops this
down significantly to a few thousand per second.  OpenPGM runs at 30-120,000
packets-per-second on gigabit hardware.

I also have various papers on adaptive FEC such that the amount of redundant
data is created in proportion to the actual data loss.  There is an
underlying strategy of sources polling receivers for congestion reports to
gauge network health.  So the protocol has support and can be extended to
support adaptive encoding but the OpenPGM implementation currently does not
have production quality and so is disabled.

If you read up on other FEC strategies such as implemented by ALC/FLUTE be
aware that they reach the Shannon's channel capacity due to very long
encoding only possible in very large static pieces of data, i.e. database
distribution, and the encoding scheme is designed to withstand significant
burst dataloss which realtime streams inherently cannot.

My company ( http://miru.hk ) has only received proposal requests for
television networks to utilize FEC technology, but many exchanges provide
multicast data feeds with out-of-band or FEC recovery but it always is
bespoke designed protocols, SIX, BATS, NASDAQ, etc.

http://www.batstrading.com/resources/membership/BATS_MC_PITCH_Specification.pdf

-- 
Steve-o
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.zeromq.org/pipermail/zeromq-dev/attachments/20110104/bf015437/attachment.htm>


More information about the zeromq-dev mailing list