[zeromq-dev] Simple GFS-like System

[Wolfgang Richter]

Thanks for the interest everyone :-)

This really grew out of some frustration with performance issues (and
configuration complexity!) I was seeing using KFS and HDFS on a small
Eucalyptus cluster (inside VMs).  I've also seen performance issues on bare
metal.

I decided I wanted something dead simple that I could reason about and get
baseline numbers out of for reading and writing into a cluster.

Turns out ZeroMQ + my simple implementation on top of Redis can out perform
them in read/write performance (more evaluation I hope to put on the web in
the near future).

--
Wolf

On Sat, May 14, 2011 at 11:21 AM, Thomas S Hatch <thatch45 at gmail.com> wrote:

>
>
> On Sat, May 14, 2011 at 12:41 AM, Pieter Hintjens <ph at imatix.com> wrote:
>
>> On Sat, May 14, 2011 at 8:37 AM, Wolfgang Richter <wolf at cs.cmu.edu>
>> wrote:
>>
>> > Backed by ZeroMQ (right now, only REQ-REP sockets; planning on using
>> > PUSH-PULL for pipelining later) and Redis (for metadata storage):
>> > https://github.com/theonewolf/TripleD/
>>
>> Neat! I've sent a tweet to my three loyal followers.
>>
>> It's really great to see ZeroMQ being used as the transport layer for
>> smart new projects like this. If you want maximum visibility for
>> Triple-D, I'd suggest writing a blog posting that explains your goals,
>> and also acts as a simple tutorial for ZeroMQ.
>>
>> -Pieter
>> _______________________________________________
>> zeromq-dev mailing list
>> zeromq-dev at lists.zeromq.org
>> http://lists.zeromq.org/mailman/listinfo/zeromq-dev
>>
>
> This is a great concept! I use MooseFS a great deal in my datacenters and
> over and over again I say "we need this written with ZeroMQ and to use redis
> as the metadata back end"!  You are on the right track, and I hope to see
> this grow into something more powerful!
>
> -Thomas S Hatch
>
> _______________________________________________
> zeromq-dev mailing list
> zeromq-dev at lists.zeromq.org
> http://lists.zeromq.org/mailman/listinfo/zeromq-dev
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.zeromq.org/pipermail/zeromq-dev/attachments/20110515/e55dfb7e/attachment.htm>