[zeromq-dev] 0MQ vs TCP sockets - survey

[Kelly Brock]

Hi Pieter,

> Hi y'All,
> 
> As a kind of a survey / test to see if you're awake, I'd like to
> solicit your views, as 0MQ users and contributors, about the most
> significant differences between 0MQ and traditional TCP sockets.  I.e.
> how would you convince your boss that it was worth using 0MQ rather
> than just coding your app the old fashioned way...?
> 
> Feel free to +1 / 'this' other people's comments so we know what
> tickles you the most.  I'll collect the results in a comparison
> document on the wiki.

	For the simple parts of my work I'm definitely able to sell 0mq
already and I'm actively using it.  I've wrapped up the details and hidden
everything behind some C++ to make it look a little more traditional so some
hard nosed snobs didn't bitch, but it works like a charm for a simple
distributed asset processor system.

	My initial plans for the tool in question actually had some folks
very scared about what I was doing.  It sounded a hell of a lot more
complicated than it was and 0mq actually helped sell it given the tiny and
easy to use api.  They all worried about the distributed nature of the tool,
they all worried about threading issues, they all worried it would be too
slow even when distributed, etc etc..  0mq helped in a couple places in
selling this:

1.	Given that I have to deal with around 1 million files which add up
to around 30-40 gigs of data, or near a terabyte if I had to process the
source data, going distributed was an obvious answer.  Everyone was thinking
in terms of traditional sockets and all the problems that involved.  I
showed them 0mq and how easy it was, they dropped that argument.

2.	Going distributed "and" multi-threaded, ack, once again, showing the
0mq model of single threaded "services" even if they happen to run in the
same process, helped win them over.

3.	Time involved in getting things setup.  This was even more
difficult, but they figured we need the tool so give him time.  I had it
basically functional in about a week and very little of that time was
dealing with networking or anything, it was mostly just getting the SQL
stuff setup and functional.  Needless to say, I used the extra time to clean
up a lot and make it clean and easy for our contractors to work with when we
brought them onboard.  They were very happy with it being done on time and
the lack of problems scaling up the team of people working on this.


	Now, that's the good stuff.  I have a couple complaints and I
realize some of them are repeats and you have work around solutions and
reasons you don't like them, but I'm going to repeat them anyway:

1.	Lack of connect/disconnect notification.  I know, talked about to
death, but there are simply certain patterns where it is the
cleanest/easiest and most useful way to approach notifying workers that
their task master died or has become unavailable for some reason.  Some
reasons and ways I would use this knowledge:

a)	For the task master, I would start a single worker on the local
machine and start processing.  If results are not pending for say 1 second,
bring up another worker on some other machine.  (I use zeroconf to find
other possible worker machines.)  If any worker dies/disconnects,
immediately recycle outstanding work items to other machines and potentially
bring up another worker to replace the old one.  (Down side, would need to
forcibly disconnect and ignore any outstanding data on other sockets
associated with the worker.  Part of the reason I like shared sockets when
head of the line is not an issue.)

b)	For a worker, any disconnection from the task master causes an
immediate shutdown as there is no purpose to exist anymore.  Again, because
I use zeroconf and migrate work where ever their may be free CPU/memory,
this just makes sense for my distribution model.  I share a group of
machines which all run a little controller daemon/service with other tools
and processes; the controller advertises cpu/memory/disk/etc stats and a
task master can pick the lowest utilized machine to start a new service on
from that list.  I don't want processes sitting around in memory if the task
master dies.

2.	Transacted messages or at least "acks" and knowledge or where it
went.  Eek, I know, don't like those at all.  But, especially in the case of
the way I'm using 0mq, it would make things "sooooooo" much easier since it
is exactly what I need for this tool.  I.e. I send out a work item, I "need"
the result of the processing or it invalidates the entire run.  I currently
send the results back on a separate push channel which basically means that
any worker that fails causes the task master to invalidate the entire run
since it doesn't know what went where and how to recover and send out repeat
requests.  I "could" add some timeout logic, and probably will; it is just a
case that this seems like something 0mq should support without user side
work arounds.

3.	I would greatly prefer a reactor pattern over the current select
like behavior when dealing with multiple sockets.  I know there was some
talk of this but I have not seen much further discussion lately.


	Those are basically the only complaints I have; unfortunately they
seem pretty significant considering how the discussions go on each of them.


	As a final note:  If you are worried about adding to the header of
the basic messages in order to supply ack/transaction etc, you might
consider using a simple bit slicing scheme for the wire format.  One of the
items I was very happy with in my homebrew networking was the low overhead
but high flexibility of the packet framing.  Basically it was the following:

Total Packet Size
	7 bits per byte of "size" data, high bit designates "end".  So, 1
byte for any number lower than +-64.  I.e. small packets.  2 bytes for 16k,
etc.  The value is encoded as: "(abs( length ) << 1) | length>0 ? 0 : 1" in
order to prevent small negative values from taking 5 bytes.

Optional Data
	The "sign" of the total packet size indicated if there was an
"optional" field in the packet.  If total packet size is negative the next
byte afterwards designates an options length.  If the first byte of the
options length is terminal and negative i.e. has high and low bit set, then
it is just 6 bits of "flags" how ever you want to use them.  If not it
designates that a block of control data exists and its length.

Actual Payload
	Absolute value of "Total Packet Size" - if exists( optional data
length encoded size + absolute value of optional data length ).

	For your uses, you could modify the encoding such that the first
byte of length also encodes the "more" flag or whatever you use it for, and
that still gives you +-32 byte packages at 1 byte and the optional data
field available.


	Sorry to go on about this, you got me to thinking about my "desired
ultimate" solution, which 0mq is only partially there in my opinion.  Again
though, I have different requirements than what it seems 0mq was written
for.

KB