[zeromq-dev] .NET Example

[Martin Sustrik]

Hi Mitchel,

> In regard to your comment about writing to a database. That is one of 
> the major features I was looking for in queue management software. I do 
> not want to keep track of and maintain a database schema at every host I 
> have set up to run a piece of my service.

Ok. Understood.

> I will be using the queue software to distribute the load of processing 
> a task across many servers... I.e.
> 
> Step 1. Receive and store audio file from user on server A
> Step 2. Send file to server B to be converted to a different format.
> Step 3. Send file to server C to have it delivered via e-mail.
> Step 4. Send message to server D to have the meta data stored in a 
> relational database table.
> 
> Now you can see the problem if I were to store the file in an outbound 
> queue and the server were to lose power. I can't have a file disappear 
> from within the system. So in my application some sort of fault tolerant 
> mechanisms should be used here.

Few scenarios for fault tolerance:

1. Failure because of bug in your program: Fix the bug.
2. Power outage: Buy an UPS.
3. Disk failure. Buy a redundant disk storage.
4. Box destroyed: Build a cluster.
5. Datacenter nuked: Build two geographically separated datacenters.

Side note: If you believe your application is power-failure-tolerant, it 
probably isn't. I won't go into detail, however, during the power 
failure some pieces of hardware die more quickly than others. One 
unfortunate consequence is that HD can still write data while the memory 
it's fetching the data from is already dead. This causes random damage 
to the data on the disk. Enterprise-level databases shield you from the 
damage using clever journaling techniques, but it's not 100% safe either.

So my advice is: If you value your data, buy an UPS. Also, save your 
data to the disk before UPS shuts the system down.

> I really appreciate your reply but it looks like in my case I will have 
> to look at another queue software that has this sort of feature.
> 
> ØMQ does look like a good project, and one feature that specifically 
> caught my eye was the amount of message per second and the sending rate 
> was in the megabits.. or was it gigabits?!!! made it really impressive.

The highest rate we've seen so far was ~4Gbits/second for 64kB messages 
(on InfiniBand). The rate can be presumably higher for larger messages. 
It would be interesting to explore this area and possibly build in some 
copy-offload techniques for very large messages.

> I will keep ØMQ in mind for a future project.

Sure :) Take care.

Martin

> Thank you,
> Mitchel Constantin
> 
> On Sat, Apr 25, 2009 at 1:25 AM, Martin Sustrik <sustrik at fastmq.com 
> <mailto:sustrik at fastmq.com>> wrote:
> 
>     Hi Mitchel,
> 
> 
>         Congrats on the product, ØMQ looks really neat.
> 
>         I've been evaluating messaging systems for the last week and
>         would really like to try out ØMQ.
> 
>         Are there any C# or .NET examples available?
> 
> 
>     Have a look at performance tests (cs_local_thr.cs, cs_remote_thr.cs
>     etc.)
> 
>     Use C++ man page for detailed info on the API. Function names,
>     parameter names and semantics are the same for all languages:
> 
>     http://www.zeromq.org/local--files/area:docs-v06/api_thread_t.pdf
> 
> 
>         I did add libclrzmq.dll to one of my .NET projects. However, I
>         couldn't figure out what the next step was.
> 
>         Does ØMQ offer persistent storage for messages? (Basically, to
>         CYA in the event that a server went down and came back up.)
> 
> 
>     0MQ doesn't have a server. It uses "brokerless" model. Thus you can
>     choose either to communicate directly between 2 applications or you
>     can write your own mini-broker. Have a look at "chat" example. The
>     "chatroom" component is actually a mini-broker consisting of a
>     single queue.
> 
>     As for persistent messages, no, they are not supported at the moment
>     (although on-disk offload of messages is supported). You'll have to
>     write your messages into database by hand. However, note that
>     storing message into database decreases performance by approximately
>     two orders of magnitude, so you should decide whether it's worth of it.
> 
>     Hope this helps.
>     Martin
>