[zeromq-dev] Multicast and logging

[Guido Goldstein]

Hi all!

Today we've had a discussion on IRC about when one would use IP
multicast and when better not.

The idea was to have a bus-like structure w/ multiple senders and
receivers (PUB/SUB over PGM/EPGM).

Such a structure will become very complicated when using TCP.
And this complexity *will* show up in the performance!
Every publisher has to have a TCP connection with every subscriber.
A lot of connections... all transporting the same data... FAST please!
so the idea was to use IP multicast as a transport.

The use-case for this has been distributed logging, where lots of
machines send their logging messages and an unknown number of consumers
do receive and process them.

I tossed in that most larger sites don't use log-levels anymore and so
you have *lots* of messages flying around. Using IP multicast does help
here because it duplicats network packets only when needed and also
very fast.

Of course, everything is sent over the network in this model.
This might seem like a waste of resources...

OTOH, consider the following questions:   [1]
  1. when do you raise the log-levels?
  2. does raising the log-levels change the behaviour of the system?

To shorten the discussino on IRC, I've been asked to provide some
information on that topic.
So here some links from which I gathered my immense knowledge:   :-)

  1. http://highscalability.com/blog/2007/8/30/log-everything-all-the-time.html
     How it all began.
  2. http://highscalability.com/blog/2008/11/24/product-scribe-facebooks-scalable-logging-system.html
     If you want so see something working.
  3. http://www.amazon.com/Scalable-Internet-Architectures-Theo-Schlossnagle/dp/067232699X/
     You've read that, right? :)

Enjoy! Hope you'll get the picture.

Cheers,
  Guido

[1] Answers:
  1. When it's to late.
  2. Yes.