[zeromq-dev] multiple bind with one socket
Martin Sustrik
sustrik at 250bpm.com
Sat Oct 2 08:30:26 CEST 2010
Hi Kelly,
>> How come multiple sockets use more bandwidth than a single one?
>
> Well, this is a pretty large subject and could take up considerable
> time, so I'll just generalize at a very high level. All said and done, a
> standard tcpip socket has about 1% wastage due to per packet headers. I
> believe (haven't refreshed on this in a while) it is 12 bytes of ip4 routing
> information and then the addition of 16 bytes (not including variable length
> option flags) tcp flow control data. The actual wastage value goes up or
> down based on the MTU and size of packets being sent but I'll just use 1%
> while discussing it here. I'm also going to the read buffer is always
> immediately emptied and the write buffer is always full on any socket in
> order to keep things simple.
>
> With 2 separate sockets you have 2% total wastage because each
> socket gets its own set of packets each with unique header information.
> This can vary even further since the network stack can decide to allocate
> packets in various ways to balance between the connections. For instance,
> if the network stack believes latency is important it could simply half the
> size of the packets it sends for each of the connections such that each
> socket is writing less data but more often. Of course this can easily bump
> the waste up to 1.25-1.5% per connection for 3-4% overhead total. Or, the
> stack could use the same size max MTU packets just alternating between the
> sockets so per second the wastage is only 1% but throughput is reduced by
> the additional 1% overhead in the additional header stream, but latency is
> going to effectively double.
>
> I imagine folks will bring up header compression and other
> optimizations, new hardware features etc etc. You can discuss such things
> till everyone is blue in the face and the only absolute guarantee is the
> hardware you have where ever you are testing. When you go to the internet
> all bets are off. As soon as you run into an older router, not "if" you run
> into one, your limitations are defined by that router no matter if both
> sides of it are fiber.
>
> There are hundreds of permutations of how sockets interact with each
> other. It is nearly impossible to guarantee anything as absolute due to
> variations in anything from the local nic chip buffer size, to how long one
> network cable is from router to router. Can I say that "generally"
> combining the connections is a win, for 2, maybe not, 3 or more I think I
> can say "generally" yes.
Thanks for extensive explanation!
Now consider this:
1. If the packets are full (reach MTU size) the bandwidth overhead is
same irrespective of how many TCP connections there are (it's IP traffic
after all).
2. If packets are smaller, the overhead goes up. If you send just one
byte, there 2000+% overhead. However, this is done to improve latency --
i.e. we don't wait for more data to arrive from user, and send straight
ahead -- which is a good thing.
3. If there are many TCP connections, each sending small amount of data,
thus creating large overhead (as in 2) which, combined, hit the network
bandwidth limit, TCP CC is going to hit in and senders will back off.
Backing off in turns means accumulating more data in each packet until,
ultimately, optimal balance between bandwidth and latency is reached.
In summary, the whole thing works irrespectively of number of
connections involved. Additional bandwidth is used only if it's
available and as a means to improve latency.
If you want to explicitly trade latency for bandwidth, we can allow
turning nagle on (it's off by default for 0MQ TCP transport).
Martin
More information about the zeromq-dev
mailing list