Debugging Playbook
The previous four pages each cover one tool in depth. This one runs the other direction: given a symptom, which tool tells you the answer fastest, and what is that symptom usually a sign of. Treat it as a lookup table, not reading material - come back to it when something is actually broken.
| Symptom | Check with | Likely cause |
|---|---|---|
| Works locally, fails in prod | curl --resolve against the prod IP directly (ss, netstat & curl); dig @<prod-resolver> vs local (dig and DNS Debugging) | Config/environment drift (different DNS record, different security group, different TLS cert) rather than a code bug - if --resolve against prod's IP works from your machine, the app code is fine |
| Intermittent timeouts | mtr over several minutes, not a single traceroute (ping, traceroute & mtr) | Loss on one specific hop that a one-shot traceroute would likely miss; a link flapping or overloaded upstream |
| Connection refused | ss -tan / netstat -tan on the target for a LISTEN entry on that port (ss, netstat & curl) | Nothing is listening on that port (service down, crashed, or bound to the wrong interface) - the OS actively rejects with a TCP RST the instant the SYN arrives |
| Connection reset (mid-connection, not on connect) | tcpdump 'tcp[13]&4!=0' to catch the RST in context (tcpdump and Wireshark) | Peer process crashed or hit a timeout, or a stateful firewall/load balancer killed an idle or invalid connection - different from refused, which happens instantly on connect |
| High latency to one region only | mtr to that region specifically, compare against a working region (ping, traceroute & mtr) | A specific transit link/peering path is congested or suboptimally routed; not a global capacity problem if other regions are fine |
| DNS resolves to the wrong IP | dig @8.8.8.8 / dig @1.1.1.1 vs local resolver; dig +trace for delegation (dig and DNS Debugging) | Stale cached answer (TTL not yet expired) or split-horizon DNS intentionally returning a different answer on this network |
| TLS handshake failure | curl -v for the handshake transcript; curl -w time_appconnect - time_connect for where time is going (ss, netstat & curl) | Expired/mismatched certificate, SNI mismatch, unsupported TLS version/cipher, or clock skew on one side invalidating cert validity dates |
| One service can't reach another in the same VPC | ping/traceroute inside the VPC first to rule out routing (ping, traceroute & mtr); ss -tan on the target for LISTEN (ss, netstat & curl); tcpdump on both ends to see whether the SYN even arrives (tcpdump and Wireshark) | Security group / firewall rule blocking the port (SYN never arrives at the target's capture), or the service is listening on 127.0.0.1 instead of the VPC interface (SYN arrives but gets refused) |
Reading the table
The three columns are meant to be followed left to right under time pressure:
- Symptom is what got reported to you, usually secondhand and vague.
- Check with is the fastest command that turns the vague symptom into a concrete fact - a loss percentage, a connection state, a TLS error string.
- Likely cause is what that fact usually means, but it is a prior, not a verdict - confirm before acting on it.
When a symptom could be several of these at once (e.g. "one service can't
reach another" could be routing, firewall, or the service not listening),
work from the network layer up: confirm the path exists first (ping/mtr),
then confirm something is listening (ss), then look at what's actually
happening at the packet level if the first two don't explain it (tcpdump).
Jumping straight to tcpdump on a routing problem just shows you no packets
arriving, which mtr would have told you faster.
"Connection refused" and "connection reset" get used interchangeably in bug
reports but mean opposite things about when the failure happened: refused
is instant, on the very first SYN, because nothing is listening; reset can
happen at any point in an established connection because something actively
tore it down. Always ask (or check ss/tcpdump) which one it actually was
before guessing at a cause.
Start at this table, run the one "check with" command it points to, and use the result to decide whether to keep debugging yourself or hand off with a concrete fact ("SYN never reaches the target, security group problem") rather than the original vague symptom.