[zeromq-dev] multiple bind with one socket

[Kelly Brock]

Hi Martin,

> Hi Kelly,
> 
> >> The rationale here is that two distinct message flows should not share
> >> single TCP connection as it would result in so called head-of-line
> >> blocking, i.e. large message being transferred in one message flow
> could
> >> block the other flow for extended period of time.
> >
> > 	I understand this reasoning fine.  But, for some of my purposes, I'm
> > more concerned about multiple sockets and the bandwidth required for the
> tcp
> > flow control as I scale up the number of connections.  By design, 0mq
> seems
> > to encourage a considerable number of sockets, which is fine, but if you
> are
> > balancing bandwidth versus latency etc, sometimes merging the sockets
> can be
> > a large win.  This is especially true as wan connections are considered.
> 
> Can you be more specific about the scenario?
> 
> How come multiple sockets use more bandwidth than a single one?

	Well, this is a pretty large subject and could take up considerable
time, so I'll just generalize at a very high level.  All said and done, a
standard tcpip socket has about 1% wastage due to per packet headers.  I
believe (haven't refreshed on this in a while) it is 12 bytes of ip4 routing
information and then the addition of 16 bytes (not including variable length
option flags) tcp flow control data.  The actual wastage value goes up or
down based on the MTU and size of packets being sent but I'll just use 1%
while discussing it here.  I'm also going to the read buffer is always
immediately emptied and the write buffer is always full on any socket in
order to keep things simple.

	With 2 separate sockets you have 2% total wastage because each
socket gets its own set of packets each with unique header information.
This can vary even further since the network stack can decide to allocate
packets in various ways to balance between the connections.  For instance,
if the network stack believes latency is important it could simply half the
size of the packets it sends for each of the connections such that each
socket is writing less data but more often.  Of course this can easily bump
the waste up to 1.25-1.5% per connection for 3-4% overhead total.  Or, the
stack could use the same size max MTU packets just alternating between the
sockets so per second the wastage is only 1% but throughput is reduced by
the additional 1% overhead in the additional header stream, but latency is
going to effectively double.

	I imagine folks will bring up header compression and other
optimizations, new hardware features etc etc.  You can discuss such things
till everyone is blue in the face and the only absolute guarantee is the
hardware you have where ever you are testing.  When you go to the internet
all bets are off.  As soon as you run into an older router, not "if" you run
into one, your limitations are defined by that router no matter if both
sides of it are fiber.

	There are hundreds of permutations of how sockets interact with each
other.  It is nearly impossible to guarantee anything as absolute due to
variations in anything from the local nic chip buffer size, to how long one
network cable is from router to router.  Can I say that "generally"
combining the connections is a win, for 2, maybe not, 3 or more I think I
can say "generally" yes.

KB