[zeromq-dev] Reliability question
steven.mccoy at miru.hk
Mon Aug 2 10:49:08 CEST 2010
On 2 August 2010 16:28, Martin Sustrik <sustrik at 250bpm.com> wrote:
> The LBM solution makes following assumption: We have enough resources
> (disk space) to store the historical feed till the crybaby consumer is
> fixed/replaced/killed by the datacenter staff.
> When applying it to Internet there are two problems:
> 1. Slow/hung-up consumers are out of publisher's control. They may never
> be fixed. They can literally sit there and cause problems for years.
> 2. The resource (memory/disk space) allocated to your communication at a
> middle box (think of an Internet backbone) is going to be severely
> limited. The worst-case assumption should be that the consumer can stop
> consuming only for a fraction of a second, otherwise the buffers at the
> middle nodes start overflowing.
Which makes, just like last value caching, it more suited to a higher layer.
You need an entirely new infrastructure to propagate live data to a
continuous archive system that can scale on demand to client requests.
Otherwise you are ending up with a framework like Apache Hadoop tuned to
be equivalent of Vhayu's historical tick-data store.
-------------- next part --------------
An HTML attachment was scrubbed...
More information about the zeromq-dev