Why Is My TCP Transfer Slow?
- A single TCP connection's speed is roughly the minimum of:
rwnd / RTT,cwnd / RTT, the loss ceiling1.22 * MSS / (RTT * sqrt(p)), the link rate, and how fast the application feeds it. - In-flight bytes = throughput x latency (the BDP). A window smaller than the BDP leaves the pipe empty: 1 Gbit/s x 80 ms = 10 MB, and a 64 KB window fills 0.65% of it.
- A new connection pays a handshake, a TLS handshake and slow start before bandwidth matters: a 100 KB fetch over TCP + TLS 1.3 takes about 5 RTTs. Small objects are latency-bound, not bandwidth-bound.
- To diagnose, ask in order: is the app the limit, then
rwnd, then the send buffer, then loss, thencwnd.ss -tion the sender answers most of it. - Fixes follow the limit: scaling and buffers for windows, BBR or a shorter RTT for loss, connection reuse for cold starts, parallel streams when one flow's cap is the problem.
This page is a synthesis. The mechanisms live on their owner pages and are linked, not repeated: flow control (rwnd, scaling, buffers), congestion control (cwnd, slow start, CUBIC, BBR), retransmission and timers (RTO, fast retransmit), TCP connection (the handshake) and TLS (handshake round trips). Here the job is to combine them and answer one question: where do I look?
Mental model
Picture a water pipe between two towns. How much water the pipe holds is its width times its length. To keep water arriving at full rate, you must keep the whole pipe full: pour continuously, and by the time the first drop reaches the far end, the pipe is already packed behind it. If you pour from a straw-sized funnel, the pipe is mostly air no matter how wide it is, and a wider pipe will not help. Opening the pipe (the handshake) and slowly turning the tap up (slow start) take time before the flow is steady.
Where it breaks: water is never lost and re-sent, and the tap does not turn itself down when it senses the pipe overflowing. TCP both loses packets and cuts its own rate when it sees loss, so the "pipe" it is allowed to fill keeps resizing.
This is Little's law for a network: items in the system = arrival rate x time in the system. Here, in-flight bytes = throughput x round-trip time. It is the same arithmetic you use to size a thread pool or a connection pool: to sustain 1,000 requests per second at 50 ms each you need 50 in flight. A TCP window is the "pool size" for bytes, and throughput is window / RTT.
Where it breaks: Little's law describes a steady state, and a TCP flow starts empty and ramps up (slow start). Also, queueing in a router buffer adds to RTT, so filling the pipe past its BDP does not add throughput; it adds delay (see bufferbloat in congestion control).
How it works
The limits on one connection
A sender may have at most min(rwnd, cwnd) unacknowledged bytes in flight, and one window is delivered per round trip. That gives a rate of window / RTT. Six things can bind, and only some of them are TCP's:
| Limit | Formula | Symptom | Fix | Owner page |
|---|---|---|---|---|
| The application | rate the app writes or reads | app_limited in ss; sender idle between writes; low busy time | fix the producer or consumer (disk, database, a slow read loop) | the application |
| Receive window | rwnd / RTT | large rwnd_limited share; capture shows a full or zero window | window scaling, tcp_rmem maximum, make the reader faster | flow control |
| Send buffer | sndbuf / RTT | large sndbuf_limited share | raise the tcp_wmem maximum, avoid a small fixed SO_SNDBUF | flow control |
| Congestion window | cwnd / RTT | young connection, or cwnd far below the BDP | reuse connections, larger initial window, a better controller | congestion control |
| Loss | 1.22 * MSS / (RTT * sqrt(p)) | retrans and lost rising; rate falls as loss rises | remove the loss, shorten the RTT, try BBR | congestion control |
| Link rate | capacity of the slowest link | delivery_rate flat at line rate | a faster link (only helps when this is the limit) | latency and capacity numbers |
The loss ceiling is not independent of cwnd. It is what a Reno-like cwnd averages out to when losses arrive at rate p (Mathis, Semke, Mahdavi and Ott, 1997); it is listed separately because you compute it from a measured loss rate instead of reading cwnd off the socket. The constant 1.22 is sqrt(3/2), and it is an approximation that depends on the loss model and on delayed ACKs.
Put together, one connection reaches about:
throughput = min( rwnd / RTT, cwnd / RTT, 1.22 * MSS / (RTT * sqrt(p)), link rate, app rate )
Whichever term is smallest is the limit. Fixing any other term changes nothing until the smallest one moves, which is why measuring before tuning matters.
Worked example: the pipe and the straw
The bandwidth-delay product (BDP) is bandwidth x RTT, the number of bytes that must be in flight to keep the link busy (flow control derives it for a 100 Mbit/s path). Take 1 Gbit/s and 80 ms:
- BDP = 1,000,000,000 bit/s x 0.08 s = 80,000,000 bit = 10,000,000 bytes, 10 MB.
- A 65,536 byte window sends one window per RTT: 65,536 B / 0.08 s = 819,200 B/s = 6.55 Mbit/s, which is 0.655% of the link.
- To fill the pipe the window must be at least 10,000,000 bytes. The 16-bit field tops out at 65,535, so you need a window scale shift
swith65,535 x 2s >= 10,000,000:2s >= 152.6, sos = 8. The protocol allows up to 14 (RFC 7323 section 2).
Worked example: the loss ceiling
Take MSS 1460 bytes (11,680 bits), RTT 50 ms and loss 0.1% (p = 0.001):
MSS / RTT= 11,680 bit / 0.05 s = 233,600 bit/s.sqrt(0.001)= 0.03162.- Rate = 233,600 x 1.22 / 0.03162 = 284,992 / 0.03162 = about 9.0 Mbit/s, which is about 1.13 MB/s.
That figure holds on a 100 Mbit/s link and on a 10 Gbit/s link. Halving the RTT doubles it; a 100 times higher loss rate cuts it by 10 (congestion control has the loss table). The formula is a model of one Reno-like flow in steady state; CUBIC and BBR do better or worse depending on the path.
Worked example: what a cold connection costs
A client that has never talked to this server fetches a page over TCP and TLS 1.3. Before the first byte of the response can arrive, it pays:
- 1 RTT for the TCP handshake (TCP connection).
- 1 RTT for the TLS 1.3 handshake; the client's request leaves with its
Finishedmessage (TLS: what a round trip costs). - Slow start for the response. The initial window is 10 segments:
IW = min(10 * MSS, max(2 * MSS, 14600))(RFC 6928), which is 14,600 bytes for MSS 1460. The server may send 10, then 20, then 40 segments, one doubling per RTT, and the first round of data arrives one RTT after the request leaves (congestion control).
How the table below is derived: segments = ceil(bytes / 1460); slow-start rounds are the smallest n with 10 * (2n - 1) >= segments (cumulative 10, 30, 70, 150, 310, 630, 1270); time to last byte = (2 + rounds) * RTT. A KB here is 1,000 bytes.
| Object | Segments | Rounds | RTTs to last byte | RTT 10 ms | RTT 50 ms | RTT 150 ms |
|---|---|---|---|---|---|---|
| 15 KB | 11 | 2 | 4 | 40 ms | 200 ms | 600 ms |
| 100 KB | 69 | 3 | 5 | 50 ms | 250 ms | 750 ms |
| 1 MB | 685 | 7 | 9 | 90 ms | 450 ms | 1,350 ms |
Read it as a floor. It ignores bandwidth, so serialization time is extra, and it assumes rwnd never binds, which a 1 MB transfer will probably violate before round 7. Two readings matter:
- Without TLS (plain TCP) subtract one RTT: 100 KB at 50 ms is 4 RTTs, 200 ms. A 100 KB page costs 4 to 5 round trips before the link's speed enters the calculation.
- 15,000 bytes is just over the 14,600 byte first flight (11 segments against 10), so it needs a second round trip. The same fetch over a warm connection whose
cwndalready exceeds the object costs 1 RTT, since the request goes out and the whole response comes back. That assumes thecwndsurvived the idle gap (slow-start restart after idle).
UDP-based transports sit outside this arithmetic. UDP has no rwnd, cwnd or handshake, so none of the TCP ceilings apply to it; anything built on it (QUIC, for example) must re-implement flow and congestion control, and the application owns loss recovery. For where TCP's guarantees come from, see the transport overview.
Deep dive
Diagnosing with ss: which limit is it?
On the sending host, ask ss for the kernel's per-connection counters (ss -ti, Linux). The fields below come from struct tcp_info (include/uapi/linux/tcp.h) as printed by iproute2's ss.c; only some of them are in the ss(8) man page, and they need a reasonably recent kernel and iproute2.
| Field | What it tells you | Points at |
|---|---|---|
app_limited | the sender did not have enough data to fill the window | the application |
busy:<ms> | time the connection spent with data in flight; the denominator for the next two | reference |
rwnd_limited:<ms>(<pct>%) | share of busy time the receiver's window was the limit | receive window |
sndbuf_limited:<ms>(<pct>%) | share of busy time the send buffer was the limit | send buffer |
retrans:<cur>/<total>, lost, bytes_retrans | retransmissions now and in total | loss |
cwnd, rtt, mss, send | send is the rate cwnd and mss imply over the measured rtt | congestion window |
delivery_rate | the measured rate at which data is being delivered | what you actually get |
There is no cwnd_limited field. Busy time that is neither rwnd_limited nor sndbuf_limited was spent waiting on the congestion window (or on pacing), so by elimination a large remainder means cwnd. Compare cwnd * mss to your BDP. If the connection is young, that is slow start; if it is old and retrans climbs, that is loss.
Three practical points about reading these:
- Run it on the sender.
rwnd_limitedis measured where the data originates, so for a slow download, runsson the server (or on whichever side is sending the bytes you care about). If you only control the receiver, a capture shows the advertised window and the retransmissions (tcpdump and Wireshark). - The connection state matters first. A pile of sockets in
SYN-SENTis a connect problem and a pile inCLOSE_WAITis an application that never closes; neither is a throughput question. TCP state machine maps each state to its cause, and error messages separates a connect timeout from a slow read. - Stalls are not slowness. A transfer that runs, pauses for hundreds of milliseconds, then runs again is often an RTO, or a delayed ACK meeting Nagle (Nagle and delayed ACK; for the RTO case see retransmission and timers). Average throughput hides it; a per-second
delivery_rateor a time-sequence graph shows it.
Tuning knobs, by the limit they move
| Knob | Moves | Helps when | Detail |
|---|---|---|---|
Window scaling, tcp_rmem and tcp_wmem maximums | rwnd and send-buffer ceilings | rwnd_limited or sndbuf_limited is high on a long path | Linux buffer autotuning; see tcp(7) for the sysctls |
| CUBIC versus BBR | how cwnd reacts to loss | random loss on a long fat path; BBR models bandwidth instead of treating each loss as congestion | Tahoe, Reno, NewReno, CUBIC and BBR |
| MTU and MSS | bytes per segment | the loss ceiling is proportional to MSS; a path that cannot carry 1500 bytes wastes segments | MTU, MSS and fragmentation |
fq qdisc pacing | burstiness of sending | senders that emit line-rate bursts into a shallow buffer; BBR depends on pacing | tc-fq(8); bufferbloat and AQM |
| TSO and GSO | per-packet CPU cost | a host that is CPU-bound at high packet rates, not a window or loss problem | kernel segmentation offloads |
| Connection reuse, resumption, 0-RTT | cold-start cost | short transfers dominated by handshakes and slow start | below, and TLS session resumption |
Offloads have one visible effect: a capture on the sending host can show "packets" larger than the MTU, because segmentation happens in the NIC or late in the stack, after the capture point. That is not a bug.
Parallel streams and why they work
Each connection has its own windows and its own loss history, so N connections add N caps. With the window-limited path from the worked example, 4 streams give about 4 x 6.55 = 26.2 Mbit/s instead of 6.55. On the lossy path above, each flow gets its own 9.0 Mbit/s ceiling, so 4 flows reach about 36 Mbit/s, if the losses are independent per flow and the link has room. Each extra flow also starts with its own 10-segment initial window, so a short fetch split across flows gets more bytes in its first round trip.
Parallelism is a workaround, not a free lunch. At a shared bottleneck, N flows take about N fair shares, so you are being less fair to everyone else; and the real fixes (right-sized windows, a loss-tolerant controller) make one flow fast. S3's guidance to fetch with parallel requests (S3 performance design patterns) and iperf3 -P (iperf3 options) both lean on this.
Warm connections: pooling, resumption, 0-RTT
The cold-start cost above is paid once per connection, so the cheapest fix for small objects is to not open new ones: keep-alive and connection pools, and HTTP/2 or HTTP/3 multiplexing so many requests share one grown cwnd (HTTP/1.1 and HTTP/2). When a new connection is unavoidable, TLS 1.3 resumption saves certificate work but still costs one round trip unless 0-RTT early data is used, which is replayable and safe only for idempotent requests (TLS: session resumption and 0-RTT). QUIC merges the transport and cryptographic handshakes so a new connection needs one round trip instead of two (QUIC and HTTP/3).
Churning through short connections has a second cost: ephemeral ports sit in TIME_WAIT, and a busy client can run out (port exhaustion and TIME_WAIT). Pooling fixes both problems with one change.
Textbook vs real world
| The textbook says | Production does | Why |
|---|---|---|
| Throughput is the link bandwidth | One flow is capped by window / RTT and by loss long before the link; web objects under about 100 KB never leave slow start | the window must equal the BDP, and a new flow starts with 10 segments |
| Packet loss is why TCP is slow | Loss is one limit of five; app stalls, small receive windows and cold starts are as common | ss shows which limit binds; slow does not mean lossy |
| Opening a connection is cheap | TCP plus TLS 1.3 is 2 RTTs before the request is even answered, then slow start | this is why pools, keep-alive and QUIC exist |
| More bandwidth makes a page faster | A page of many small objects is latency-bound; a faster link barely changes it | round trips, not bytes, dominate (the cold-fetch table) |
| A speed test measures your TCP speed | Many speed tests use several parallel connections, so the number is an aggregate, not what one connection gets | per-flow caps add up (parallel streams above) |
| Pick one fixed buffer size | Cloudflare found a small fixed tcp_rmem capped throughput on high-latency links while a large one cost kernel work, and moved to autotuning | autotuning grows each buffer only as its path needs |
In production
- Google proposed raising the initial window to 10 segments after measuring web workloads (Dukkipati et al., "An Argument for Increasing TCP's Initial Congestion Window", 2010), and the IETF standardised it as RFC 6928. It is the reason the first flight is 14,600 bytes.
- Google also describes running BBR on google.com and YouTube to avoid the loss-based ceiling on long paths (BBR, ACM Queue 2016).
- Cloudflare describes capping
tcp_rmemto limit kernel memory-collapse work, finding that the cap limited throughput on high-latency links, and moving to Linux autotuning (Optimizing TCP for high WAN throughput while preserving low latency). - CDNs shorten the RTT that every ceiling divides by: the client's TCP and TLS handshakes end at a nearby edge, and the edge reuses warm connections to the origin (CDN).
- Amazon S3 recommends parallel requests and byte-range reads for throughput (S3 performance design patterns); ESnet publishes host tuning for long fat science paths (fasterdata).
- For the web-side view of slow start and the first flight, see Grigorik's High Performance Browser Networking, "Building Blocks of TCP".
History and evolution
- 1988 Jacobson's congestion avoidance adds slow start after the 1986 congestion collapses, so every new connection starts small (congestion control history).
- 1997 Mathis et al. show that a Reno-like flow's rate falls with
1 / (RTT * sqrt(p)): loss, not link speed, can cap a single connection. - 2002 RFC 3390 raises the initial window to
min(4 * MSS, max(2 * MSS, 4380)), which is 3 segments at MSS 1460: cold fetches took too many round trips. - 2010 to 2013 Google's study and RFC 6928 raise it to 10 segments, 14,600 bytes at MSS 1460.
- 2016 Google publishes BBR, a controller that does not treat every loss as congestion.
- 2018 TLS 1.3, RFC 8446, cuts a fresh handshake from two round trips to one (TLS 1.2 needs two).
- 2021 QUIC, RFC 9000, folds the transport and crypto handshakes into one round trip and removes the serialization of TCP then TLS.
- Now Linux defaults to CUBIC; IW10 and TLS 1.3 are mainstream, and HTTP/3 over QUIC is deployed by large providers. Adoption shifts, so check a current measurement instead of a frozen figure.
- Next BBRv3 and L4S (RFC 9330) aim at very low queueing delay.
Hands-on
Measure where a request spends its time, using curl's cumulative timers (curl --write-out):
This is illustrative output for a path with a 50 ms RTT; yours will differ. The timers are cumulative from the start of the command. time_connect minus time_namelookup is the TCP handshake, about one RTT (0.052). time_appconnect minus time_connect is the TLS handshake, about one more RTT. time_starttransfer minus time_appconnect is the request round trip plus server think time. time_total minus time_starttransfer is the download. If the first three dominate, you have a latency problem and more bandwidth will not help.
Check what the sender's kernel thinks (Linux):
An abbreviated, illustrative line; field order and availability vary by kernel and iproute2 version. Here rwnd_limited is 90.8% of busy time, so the receiver's window is the limit, and cwnd:44 at mss:1448 over 81.2 ms gives send of 6.3 Mbit/s (44 * 1448 * 8 / 0.0812). Ten retransmitted segments are not the problem. Next stop: flow control. macOS has no ss; use netstat -an for state and a capture for windows.
Measure a path with a synthetic transfer (iperf3):
The columns are real iperf3's; the values are illustrative. Retr is retransmissions and Cwnd the sender's window for that interval. If -P 4 roughly quadruples the one-stream number, a per-flow limit (a window or the loss ceiling) is binding. If it does not, the shared link or the application is. Use -R to test the direction you actually care about.
Look at the loss and the path:
Loss that starts at a hop and persists to the end is real; loss at one hop that vanishes downstream is usually ICMP rate limiting (ping and mtr). For per-packet evidence, capture with tcpdump -w and open Wireshark's Statistics > TCP Stream Graphs (time-sequence, throughput, window scaling) to see stalls, retransmissions and window limits (tcpdump and Wireshark). The debugging by layer strings the tools together, and ss, nc, lsof and ip covers the socket side.
Gotchas
Interview questions
Show answer
I treat throughput as the minimum of several limits and find which one binds. First, is the application the limit, meaning it writes or reads in bursts. Then on the sender, ss -ti shows rwnd_limited and sndbuf_limited shares of busy time, retrans for loss, and cwnd against the BDP for the congestion window. The remainder after rwnd and sndbuf is the congestion window. A capture adds the advertised window and stalls. I fix only the limit that binds.
Follow-ups:
- How do you tell loss from a small window?
retransis flat for a window limit and rising for loss; a window limit gives throughput equal towindow / RTTwith no retransmissions. - Where do you run
ss? On the sender, because the limit counters are the sender's view. - What if everything looks fine? Check the application and connection setup cost, since a cold connection is 2 RTTs plus slow start.
Show answer
One window is delivered per RTT, so throughput is window / RTT. With a 65,536 byte window that is 655,360 bytes per second, which is 5.24 Mbit/s; rounding 64 KB to 64,000 bytes gives 640,000 B/s, 5.12 Mbit/s. Either way it is about 5 Mbit/s on any link, because the window, not the bandwidth, is the limit.
Follow-ups:
- How do you raise it? Window scaling and larger buffers, so the window reaches the BDP.
- What window fills a 1 Gbit/s link at 100 ms? The BDP: 1,000,000,000 x 0.1 / 8 = 12.5 MB.
Show answer
BDP is bandwidth times round-trip time, the number of bytes that must be in flight to keep a link full. It matters because a sender can only have a window of unacknowledged data outstanding; if the window is smaller than the BDP, the pipe is partly empty and throughput is window / RTT. For 1 Gbit/s and 80 ms it is 10 MB, so a 64 KB window fills 0.65% of the link.
Follow-ups:
- What sets the window? The smaller of the receiver's
rwndand the sender'scwnd(and the send buffer). - Does a bigger buffer than the BDP help? No, it only adds queueing delay.
Show answer
A new connection must complete the TCP handshake (1 RTT) and, for HTTPS with TLS 1.3, the TLS handshake (1 more RTT) before the request is sent. The response then rides slow start: with an initial window of 10 segments the server sends 10, 20, 40 segments, one doubling per RTT. A 100 KB object therefore takes about 5 RTTs, 250 ms at 50 ms RTT, before bandwidth matters.
Follow-ups:
- How do you reduce it? Reuse connections, resume TLS sessions, use HTTP/2 or HTTP/3 multiplexing, move the edge closer.
- Why 10 segments? RFC 6928 sets the initial window to 10 segments, 14,600 bytes at MSS 1460.
Show answer
Each connection has its own rwnd, cwnd and loss history, so the per-flow limits add up. If one flow is capped by window / RTT or by the loss ceiling, N flows give roughly N times that, until the link or the server saturates. Each connection also starts slow start independently. The cost is fairness: N flows take about N shares of a shared bottleneck.
Follow-ups:
- When do they not help? When the link, the disk or the application is the limit.
- What is the better fix? Make one flow fast: right-size the windows, use a loss-tolerant controller.
Show answer
Every ceiling and every handshake cost is a multiple of the RTT. A CDN terminates TCP and TLS at an edge near the user, so the client's handshakes and slow-start rounds happen over a short RTT, and the edge talks to the origin over warm, pooled connections with a grown cwnd instead of cold ones. The long-haul leg is paid once and reused, not once per user.
Follow-ups:
- Why does a shorter RTT raise throughput? Both
window / RTTand the Mathis ceiling scale with1 / RTT. - What about cache misses? The edge still pays the origin RTT, but over a pooled connection.
Show answer
CUBIC treats a loss as a congestion signal and cuts cwnd by 30%, so on a long path with random loss its rate is bounded by a Mathis-like ceiling. BBR estimates the bottleneck bandwidth and minimum RTT and paces to keep about one BDP in flight, without backing off on every isolated loss, so it can sustain far higher throughput when the loss is not real congestion. It does not help a receive-window limit, and early BBR could take more than its share against loss-based flows on shallow buffers.
Follow-ups:
- Does BBR need pacing? It sends at a paced rate, which on Linux can come from the
fqqdisc or the stack's own pacing. - When is CUBIC fine? Low-RTT, low-loss paths, which includes most datacentre traffic.
Fun facts
- The first flight of a new connection is 10 x 1460 = 14,600 bytes, which is why performance guides talk about a roughly 14 KB budget for the first round trip.
- A 15,000 byte response is 400 bytes over that first flight, and those 400 bytes cost a whole extra round trip.
- 1 Gbit/s at 80 ms needs a 10 MB window, which needs a window scale shift of at least 8 (65,535 x 28 is 16.8 MB).
- Halving the RTT doubles both the window ceiling and the Mathis ceiling, which is why moving the server closer beats buying a faster link.
Cheat sheet
- window / RTT - the throughput of one flow when a window binds
- BDP - bandwidth x RTT; 1 Gbit/s x 80 ms = 10 MB
- 14,600 B - initial window, 10 segments at MSS 1460 (RFC 6928)
- 2 RTT - handshake cost of a cold TCP plus TLS 1.3 connection before the request is sent
- 5 RTT - last byte of a 100 KB response over a cold TCP plus TLS 1.3 connection
- 1.22 x MSS / (RTT x sqrt(p)) - the approximate loss ceiling; 9.0 Mbit/s at 50 ms and 0.1% loss
| Order | Question | Check | Go to |
|---|---|---|---|
| 1 | Is the application the limit? | app_limited, busy | the application |
| 2 | Is the receiver's window the limit? | rwnd_limited | flow control |
| 3 | Is the send buffer the limit? | sndbuf_limited | flow control |
| 4 | Is loss the limit? | retrans, lost, mtr | congestion control |
| 5 | Is the congestion window the limit? | cwnd against BDP | congestion control |
| 6 | Is it just a cold, short transfer? | curl -w timers | TLS, connection reuse |