[zeromq-dev] Reliability question

[Steven McCoy]

On 2 August 2010 16:28, Martin Sustrik <sustrik at 250bpm.com> wrote:

> The LBM solution makes following assumption: We have enough resources
> (disk space) to store the historical feed till the crybaby consumer is
> fixed/replaced/killed by the datacenter staff.
>
> When applying it to Internet there are two problems:
>
> 1. Slow/hung-up consumers are out of publisher's control. They may never
> be fixed. They can literally sit there and cause problems for years.
>
> 2. The resource (memory/disk space) allocated to your communication at a
> middle box (think of an Internet backbone) is going to be severely
> limited. The worst-case assumption should be that the consumer can stop
> consuming only for a fraction of a second, otherwise the buffers at the
> middle nodes start overflowing.
>
>
Which makes, just like last value caching, it more suited to a higher layer.
 You need an entirely new infrastructure to propagate live data to a
continuous archive system that can scale on demand to client requests.
 Otherwise you are ending up with a framework like Apache Hadoop tuned to
be equivalent of Vhayu's historical tick-data store.

-- 
Steve-o
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.zeromq.org/pipermail/zeromq-dev/attachments/20100802/c3e58f61/attachment.htm>