Loss, Throughput and the Bandwidth-Delay Product
- A path's throughput is at most the slowest link on it (the bottleneck). Upgrading any other link changes nothing.
- Bandwidth is capacity, throughput is what you actually move, goodput is the useful application bytes left after headers and retransmissions.
- Packets are lost for four reasons: a full buffer, bit errors, a failed link or device, and deliberate policy drops. TCP cannot tell them apart and treats loss as congestion.
- Bandwidth-delay product (BDP) = rate x RTT is the data that must be in flight to fill the pipe. 1 Gbps x 80 ms = 10 MB.
- One TCP flow with a small window cannot fill a fat, long pipe: throughput is at most
window / RTT, whatever the link rate.
Mental model
Think of water pipes joined in series: a wide main, then a narrow garden hose, then a tap. The flow at the end is set by the narrowest pipe, and replacing the wide main with a wider one does nothing. The water in the pipes at any instant is width times length: a long, wide pipe holds a lot, and you only get full flow out of the tap once the whole pipe has filled.
Where it breaks: pipes do not drop water when they are full, they push back. A router with a full buffer discards the packet, and the sender only finds out later, from a missing acknowledgement.
This is a pipeline with bounded queues between stages. Throughput equals the rate of the slowest stage, and the amount of work in flight equals rate times latency (Little's law: items in the system = arrival rate x time in the system). A sender that keeps fewer items in flight than rate x latency leaves the pipeline idle, exactly like a thread pool smaller than throughput x service time.
Where it breaks: a bounded queue in your code usually blocks the producer. A network queue that is full simply drops the newcomer, and the "producer" is a remote TCP stack that learns about it one RTT later.
How it works
Delay is owned by the Delay and latency page (queuing, transmission, propagation, RTT). This page starts from the other two ideas: what happens when a queue overflows, and how fast a stream of packets can actually go.
Why packets are lost
Every router output link has a finite buffer. Packets arrive in bursts, the link drains at a fixed rate, and when the buffer is full the next arrival is dropped. That is the common cause, and it is a congestion signal. It is not the only one:
- Buffer overflow - arrivals exceed the outgoing link rate for longer than the buffer can absorb. Fixed by sending less (congestion control) or by more capacity.
- Bit errors - noise corrupts a frame and the link-layer checksum rejects it (see error detection). Rare on fibre, much more common on radio. Wi-Fi retransmits frames at the link layer, so loss often shows up as extra delay and jitter instead of a missing packet (see Wi-Fi).
- Link or device failure - a cut cable, a rebooting router, a route that has not reconverged yet. Loss is total for the affected flows until the path heals.
- Policy drops - rate limiters (policers) and firewalls drop on purpose, and active queue management (AQM) drops early to keep queues short (RFC 7567).
The standard loss metric is a one-way loss ratio measured over a stream of test packets (RFC 7680). TCP finds out through duplicate ACKs or a timeout; the reaction (slow down, retransmit) is owned by congestion control and retransmission and timers.
The bottleneck link
A path is a chain of links in series. Each can carry at most its own rate R_i, so the whole path can carry at most
throughput <= min(R_1, R_2, ..., R_n)
The word is at most. The minimum is an upper bound: window size, RTT, loss and competing traffic can push the real number lower, never higher.
When several flows share the bottleneck they divide it. With similar RTTs TCP converges roughly to an equal split, so ten flows through a 50 Mbps bottleneck get about 5 Mbps each (the mechanism is in congestion control).
Bandwidth, throughput, goodput
| Term | Definition | Unit | What limits it |
|---|---|---|---|
| Bandwidth (capacity) | Most bits per second a link can carry | bit/s | The physical layer and the contract: fibre, radio, plan |
| Throughput | Bits per second actually delivered over a path in a given interval | bit/s | The bottleneck link, competing traffic, window / RTT, loss |
| Goodput | Throughput of useful application bytes only: no headers, no retransmitted copies | bit/s | Everything above, plus header overhead and retransmission |
Headers are the first thing that lowers goodput. A full-size TCP segment over Ethernet carries 1,460 payload bytes inside a 1,500-byte IP packet (20 B IP + 20 B TCP). On the wire Ethernet adds 14 B header, 4 B frame check sequence, 8 B preamble and 12 B inter-frame gap, so one segment occupies 1,500 + 38 = 1,538 byte times:
- Overhead per segment = 40 + 38 = 78 bytes.
- Goodput fraction = 1,460 / 1,538 = 94.9%.
- On a 100 Mbps link that is at most 94.9 Mbps of goodput; on 1 Gbps, 949 Mbps. TCP timestamps (12 B per segment) cut it to 1,448 / 1,538 = 94.1%.
Retransmissions lower it further: a byte sent twice crosses the link twice and counts once. Layouts of the headers are in header diagrams; the Ethernet framing sizes are in Ethernet frames and MAC addresses.
Worked example: bandwidth-delay product
The bandwidth-delay product is the rate times the round-trip time (RTT, defined on the Delay and latency page): the bytes in flight when the pipe is exactly full. Take 1 Gbps and an 80 ms RTT:
- BDP = 1,000,000,000 bit/s x 0.080 s = 80,000,000 bits = 10,000,000 bytes, 10 MB.
- That is 10,000,000 / 1,500 = about 6,667 full-size packets "in the wire" at once.
- A sender with the classic 16-bit window, 65,535 bytes, can send one window per RTT: 65,535 x 8 / 0.080 s = 6.55 Mbit/s, which is 0.66% of the link. (65,536 B, a round 64 KiB, gives the same 6.55 to three digits.)
To fill the pipe the window must reach 10,000,000 bytes, far past what 16 bits can say. Window scaling (RFC 7323, which replaced RFC 1323) multiplies the window by 2shift with a shift of up to 14: shift 7 gives 65,535 x 128 = 8,388,480 B, still short; shift 8 gives 16,776,960 B, enough. The option mechanics and receive-buffer sizing are owned by flow control; the same arithmetic for other paths is in latency and capacity numbers.
| Path | Rate | RTT | BDP (rate x RTT) |
|---|---|---|---|
| Same data-centre rack (assumed RTT) | 1 Gbps | 0.5 ms | 62,500 B, about 61 KiB |
| Cross-country (assumed RTT) | 1 Gbps | 70 ms | 8.75 MB |
| Transpacific (assumed RTT) | 10 Gbps | 100 ms | 125 MB |
| Geostationary satellite (assumed RTT) | 50 Mbps | 600 ms | 3.75 MB |
The RTTs are round illustrative values chosen for the arithmetic, not measurements; measured RTTs are on the Delay and latency page and in the reference numbers. A path with a large BDP is called a long fat network (LFN, "elephant").
Try it below: set link rates to see the bottleneck, then pick a path and move the sender window against the BDP.
Deep dive
How loss caps a single flow: the Mathis estimate
Even with a big enough window, loss limits a Reno-style TCP flow. Mathis, Semke, Mahdavi and Ott found a steady-state rate of about (MSS / RTT) x C / sqrt(p) with C near 1.22 and p the loss probability (Mathis et al., 1997). The derivation is owned by congestion control; the Internet-level reading is what matters here. With MSS 1,460 B (11,680 bits) and RTT 80 ms, MSS / RTT is 146,000 bit/s:
| Loss rate p | sqrt(p) | Rate = 146,000 x 1.22 / sqrt(p) |
|---|---|---|
| 1% (1e-2) | 0.1 | about 1.78 Mbit/s |
| 0.01% (1e-4) | 0.01 | about 17.8 Mbit/s |
| 0.0001% (1e-6) | 0.001 | about 178 Mbit/s |
To reach 1 Gbps this way you would need p = (146,000 x 1.22 / 1,000,000,000)2, about 3.2e-8, one loss in roughly 31 million packets. The estimate does not contain the link rate at all, so a fatter pipe does not raise the ceiling: lower the RTT, lower the loss, or use a different congestion controller.
Loss versus jitter
Loss removes a packet; jitter is variation in the delay of the packets that do arrive. Queues cause both: a filling queue first adds variable queuing delay (jitter), then overflows (loss). Wi-Fi link-layer retries turn would-be loss into jitter. Real-time media cares about both: a late packet is as useless as a lost one.
Wi-Fi loss versus congestion loss
A transport sender sees only a missing ACK. Congestion loss should make it back off; random radio loss should not, because sending slower does not fix noise. Loss-based TCP backs off for both, which is why a lossy last hop underperforms its raw rate. The fixes are at the link layer (retries) and in model-based congestion control such as BBR (see In production).
Measuring honestly
Two tools answer different questions. ping sends ICMP echoes and reports loss percent plus RTT, a view of the path's loss and latency. iperf3 moves real TCP or UDP data and reports throughput. Run iperf3 -P 4 to open four parallel streams: if the sum grows, the limit was a per-flow window or loss ceiling, not the pipe. Run iperf3 -R to reverse direction (server sends), because uplink and downlink often differ. RFC 6349 describes a framework for TCP throughput testing built on exactly this: compute the ideal window from BDP, then compare the measured rate to the ideal (RFC 6349).
Textbook vs real world
| The textbook says | Production does | Why |
|---|---|---|
| Throughput = the minimum link rate on the path | That is an upper bound; the measured rate is usually lower | Window size, RTT, loss, competing flows and framing overhead all sit below the minimum link rate |
| Loss means something is broken | Some loss is deliberate: AQM drops early, policers drop above a contract rate | Drops are the only way to tell a sender to slow down; zero loss means the queue grew deep instead (bufferbloat) |
| A 100 Mbps plan means 100 Mbps downloads | Providers sell an "up to" rate; the measured number is goodput over one path to one server | Headers cost about 5%, Wi-Fi and the server are bottlenecks too, and one flow may be window-limited by RTT |
| Loss is the congestion signal | BBR and ECN-based schemes do not wait for loss | Loss arrives only after the buffer is full, which means the delay has already grown |
In production
- Google describes BBR as a congestion controller built on a model of the bottleneck bandwidth and the minimum RTT, aiming for about one BDP in flight instead of waiting for loss. The 2016 ACM Queue paper reports it deployed on Google's B4 WAN and on YouTube and Google.com (Cardwell et al., BBR, ACM Queue 2016).
- ESnet publishes tuning guidance for long, fast science-data transfers built on sizing socket buffers to the path's bandwidth-delay product (ESnet fasterdata: Linux host tuning).
- Data centre versus WAN. Inside one data centre RTTs are fractions of a millisecond, so the BDP is tens of kilobytes and a default window suffices (row one of the table above). Replication between distant regions is the LFN case: the BDP is megabytes and buffers, scaling and the congestion controller all have to be right. This is the same arithmetic, two very different regimes.
History and evolution
- 1984 John Nagle describes congestion collapse in IP networks: retransmissions of already-delivered data fill gateways and useful throughput collapses (RFC 896).
- 1988 Van Jacobson's Congestion Avoidance and Control adds slow start and congestion avoidance after real collapses, making loss a signal that senders must obey. The paper reports one LBL to UC Berkeley path falling from 32 Kbps to 40 bps (Jacobson, SIGCOMM 1988).
- 1992 Fast long paths expose the 16-bit window: RFC 1323 adds window scaling for long fat networks. RFC 7323 (2014) is the current revision.
- 1997 Mathis et al. give the first widely used loss-to-throughput formula for TCP.
- 2016 Google's BBR models the bottleneck bandwidth and RTT directly instead of reading loss as the signal.
- Now L4S (RFC 9330) combines ECN marking with a low-latency queue so senders get a congestion signal without waiting for drops (RFC 9330).
- Next Wider deployment of ECN-based signalling and model-based controllers, so loss stays rare and queues stay short.
Hands-on
Measure throughput between two hosts you control. iperf3 -s runs on the server.
This is abbreviated and illustrative, an 80 ms RTT path stuck on a 64 KiB window. Compare the bitrate to window / RTT from above. With -P 4 the [SUM] line should rise toward four times that figure if the window was the limit. -R makes the server send, so you measure the download direction. The Retr column counts retransmitted segments on the sender.
Measure loss with ping, then check where it appears with mtr:
One lost packet in 100 is 1% loss, and with only 100 samples the resolution is 1%. Run more to see small rates. A 0.2 s interval is allowed to ordinary users on Linux; macOS and some other systems require root for intervals under their minimum. In mtr, only loss that persists to the final hop counts; see ping and mtr and traceroute.
On Linux, read the live window state of a flow with ss, and the buffer limits with sysctl:
Abbreviated, illustrative values. rtt is smoothed RTT then variance in ms; cwnd is in segments, so cwnd x mss / rtt is the rate that window allows; delivery_rate is the kernel's own throughput estimate. The three tcp_rmem numbers are the minimum, default and maximum receive buffer in bytes, and the maximum should be at least the BDP (see flow control and the kernel ip-sysctl documentation). macOS has neither ss nor tcp_rmem; use netstat -s and sysctl net.inet.tcp.
Gotchas
Interview questions
Show answer
Mostly because a router's output buffer is full and it drops the next arrival, which is congestion. Other causes are bit errors (more common on wireless), link or device failures, and deliberate drops by policers, firewalls and AQM. Loss is an ordinary part of how packet networks say "slow down".
Follow-ups:
- How does TCP find out? It sees duplicate ACKs or a retransmission timeout, because the receiver never acknowledged the missing segment.
- Can TCP tell congestion loss from bit errors? No. It treats both as congestion and backs off.
Show answer
Bandwidth is the capacity of a link. Throughput is the rate actually delivered over a path. Goodput is the useful application bytes per second, after removing headers and retransmitted copies.
Follow-ups:
- What lowers goodput? Protocol headers (about 5% for full-size TCP over Ethernet: 1,460 / 1,538 = 94.9%), retransmissions, and encryption or framing overhead.
Show answer
BDP is rate times RTT: the bytes that must be in flight to keep a link busy. If the sender's window is smaller than the BDP, the pipe sits partly empty and throughput is at most window / RTT. It is the size to which windows and socket buffers must grow on fast, long paths.
Follow-ups:
- Compute it for 1 Gbps x 80 ms. 1,000,000,000 x 0.08 = 80,000,000 bits = 10 MB.
Show answer
Several things can sit between the plan and the number you see: the bottleneck may be Wi-Fi or the server rather than the access link, one TCP flow may be window or loss limited at your RTT, and headers take about 5%. The plan is capacity, not a promise for one flow to one server.
Follow-ups:
- How would you diagnose it? Run
iperf3to a known-good server, repeat with-P 4and-R, and checkss -tifor the window and RTT. If four flows add up and one does not, the limit is per-flow. - What would you change? Use a nearer server or CDN, enable window scaling and buffers sized to the BDP, or fix the lossy hop.
Show answer
It is the slowest link on a path and it caps the path's throughput. You find it by measuring along the path, for example iperf3 against hosts at increasing distance, or by checking each link's rate and utilisation, since the bottleneck is typically the one that is full while queues build in front of it.
Follow-ups:
- Why does mtr show misleading loss? Routers rate-limit ICMP replies, so a hop can look lossy while forwarding fine; only loss that persists to the last hop is real.
Show answer
Its throughput is at most window / RTT, and on a long path the RTT is large while the default window is small, so it fills only a fraction of the BDP. Even with a big window, loss limits a loss-based controller, as in the Mathis formula. A long RTT also slows the congestion window's recovery after each loss.
Follow-ups:
- What helps? Window scaling and buffers sized to the BDP, a controller built for high BDP such as CUBIC or BBR, and parallel streams.
- Why do parallel streams help? Each has its own window and loss events, so four 64 KiB windows carry 4 x 6.55 = 26.2 Mbit/s on the 80 ms path.
Show answer
The Mathis estimate says a Reno-like flow gets about (MSS / RTT) x 1.22 / sqrt(p). Rate falls with the square root of loss, so 100 times more loss costs 10 times the speed, and it does not depend on link bandwidth. With MSS 1,460 B and RTT 80 ms, p = 1e-4 gives about 17.8 Mbit/s.
Follow-ups:
- What is the assumption? Steady-state Reno-style AIMD with random loss; CUBIC and BBR behave differently.
- What loss rate would fill 1 Gbps? About 3.2e-8 for the same MSS and RTT.
Fun facts
- A 10 Gbps link with a 100 ms RTT holds 10,000,000,000 x 0.1 / 8 = 125 MB in flight.
- 1 Gbps x 80 ms is 10 MB, about 6,667 full-size packets all travelling at once.
- Jacobson's 1988 paper reports a path whose throughput fell from 32 Kbps to 40 bps, a factor of 800 (paper).
- The maximum window scaling shift is 14, which allows a window of about 1 GiB (RFC 7323).
Cheat sheet
- BDP = rate x RTT - 1 Gbps x 80 ms = 10 MB
- Window cap - throughput is at most window / RTT; 64 KiB at 80 ms = 6.55 Mbit/s
- Goodput over Ethernet - 1,460 / 1,538 = 94.9% of line rate
- Mathis - rate about (MSS / RTT) x 1.22 / sqrt(p); 100x loss = 10x slower
- Bottleneck - path throughput is at most the minimum link rate