[zeromq-dev] multiple bind with one socket

[Kelly Brock]

Hi Martin,

> >> How come multiple sockets use more bandwidth than a single one?
> >
> > 	Well, this is a pretty large subject and could take up considerable
> > time, so I'll just generalize at a very high level.  All said and done,
> a

	Please keep in mind that this is not meant argumentatively, just
discussion.  This may simply not be appropriate for 0mq, though I still
"wish" it were. :)

> Thanks for extensive explanation!
> 
> Now consider this:
> 
> 1. If the packets are full (reach MTU size) the bandwidth overhead is
> same irrespective of how many TCP connections there are (it's IP traffic
> after all).

	Correct only in that you can't waste more in terms of "per-second"
once you've maxed out the pipe at full MTU.  It still doesn't come for free
though as you have to send the additional header data for the new connection
thus reducing your actual throughput potential.  Using nice easy numbers,
say you send 10k bytes of data and it costs 100 bytes of header data to
transmit that, so it is a total of 10100 bytes you actually transmit.  Split
the data in half and send over 2 connections, you have to send 10200 bytes
of data for the same 10k bytes because of the tcp overhead.  For each new
connection, add another 100 bytes of data to be sent.  So, no the data sent
per second doesn't change; you just have to send more data over time.

> 2. If packets are smaller, the overhead goes up. If you send just one
> byte, there 2000+% overhead. However, this is done to improve latency --
> i.e. we don't wait for more data to arrive from user, and send straight
> ahead -- which is a good thing.

	Being a good thing depends on the use case which is why I say this
could very well be beyond the scope of 0mq intentions though I personally
have a mix of requirements which cover both sides.  On one side the above
behavior is fine as I really don't care too much about the wastage at some
points.  On the other hand, there are some items which happen as discreet
messages constantly and very quickly where some amount of coalescing is
definitely a good thing because it could easily blow out even my internal
bandwidth very quickly.

> 3. If there are many TCP connections, each sending small amount of data,
> thus creating large overhead (as in 2) which, combined, hit the network
> bandwidth limit, TCP CC is going to hit in and senders will back off.
> Backing off in turns means accumulating more data in each packet until,
> ultimately, optimal balance between bandwidth and latency is reached.

	This is again, technically correct but with limitations.  Even on a
local area network congestion controls can cause problems among many sockets
connected to the same machine.  One packet collision on a router could throw
a single one of your connections into slow restart, which in turn can bring
down either machines throughput enough to back off the sliding window of
another socket, which can of course cause even lower throughput etc etc.
"Most" tcp stacks have multiple protections against the feedbacks such as
this but they almost all rely on backing off the nic level window in order
to prevent further oscillations.  This of course drops the bandwidth and
throughput of the entire machine for a bit.  A single connection does not
have as many, though it still has plenty, of ways in which it can get forced
into a reduced window and/or slow restart/exponential backoff etc etc
congestion mechanisms.

	This is not fun stuff to fix after the fact.  I spent a year on live
maintenance looking at bandwidth logs and rewriting systems to avoid
problems caused by tcp congestion control.  One of the big wins was exactly
what I mentioned, merging several sockets to reduce collision problems,
excessive bandwidth overheads and related items.  In many cases we didn't
have to do any further work as it was enough to prevent the larger
oscillation problems.  Of course, getting rid of those problems usually let
us see where someone totally screwed the pooch in a different area...  :/
If I remember correctly, this was FreeBSD 5.1 at the time; it's probably
gotten better since then.

> In summary, the whole thing works irrespectively of number of
> connections involved. Additional bandwidth is used only if it's
> available and as a means to improve latency.

	Once again, technically correct except that each connection adds to
the amount of data you have to transfer in terms of the number of headers to
send the same amount of data.  Again, keep in mind I'm maintaining a VERY
simplistic view in terms of 50:50 sharing to keep this simple, obviously
this will change based on actual connection utilization in a real app.

	Of course what this overhead does in my case is cause problems for
users on slow connections.  A fairly common usage for my old work was people
using laptops tethered to cell phones.  Those users have relatively high
bandwidth in spikes, low to medium latency for "small" things and constant
congestion problems with dropped packets etc.  Multiple connections in this
case is definitely not a good solution, but maybe that is getting too
specific for why I'd definitely prefer single connections to clients.

> If you want to explicitly trade latency for bandwidth, we can allow
> turning nagle on (it's off by default for 0MQ TCP transport).

	I'll have to run an experiment with removing the no_delay flags from
the sockets.  One of my quick experiments early on may have been affected by
that since it sends 12 bytes per item at regular (quick) intervals. With
only 1/100th of my dataset it was pushing about 200mb's a second which was
completely unacceptable.  This might be the explanation.

	Unfortunately, I'm working the weekend so won't get to it till later
at best.

KB