Most error messages are written outside-in: the component that gave up speaks first, and the thing that broke is buried in the last clause. We read them the same way we read English, left to right, and we start debugging the loudest noun in the first line. That noun is usually the victim.
Two failures taught me to read the other direction. Both cost me time I would have kept by starting at the innermost cause.
Case 1: the registry error that was never about the registry
A deployment could not pull an image:
failed to resolve reference "registry.internal.example.com/web:staging"
dial tcp: lookup registry.internal.example.com on 10.0.0.1:53:
read udp 10.0.0.55:50332->10.0.0.1:53: read: connection refusedRead the first line and the obvious suspects are the registry, the image tag, and my credentials. I nearly went and checked whether the tag had been pushed.
Read the last clause and there is no ambiguity at all:
read udp 10.0.0.55:50332 -> 10.0.0.1:53: read: connection refused
^^ ^^^^^^^^^^^^^^^^^^
DNS nothing is listeningPort 53 is DNS. connection refused means a machine answered and declined,
which is what a host does when no service is bound to that port. So: this
machine asked its configured resolver for a name, and the resolver was not
running. Docker never resolved the hostname, never opened a TCP connection,
never presented a credential. Everything the first line names had not happened
yet.
This is the registry row from the deploy pipeline writeup, and the cause was mundane. The host was configured to use the LAN gateway as its DNS server, and that gateway was not running a DNS service that accepted queries. Any name lookup on that box would fail. It surfaced as a Docker problem because a deploy happened to be the next thing that needed a name.
Diagnosis is three commands, in order, each one testing a smaller claim than the last:
cat /etc/resolv.conf # who am I asking?
dig @10.0.0.1 registry.internal.example.com # does that server answer at all?
getent hosts registry.internal.example.com # does the system resolve it?If the second one refuses, nothing above it can work, and no amount of
docker login will help.
Fixing the host resolver is the real repair. There is a Docker-level fallback
in /etc/docker/daemon.json:
{
"dns": ["1.1.1.1", "8.8.8.8"]
}Use public resolvers only if the names you need are public. Mine were internal,
so pointing at 1.1.1.1 would have swapped connection refused for NXDOMAIN and
left me debugging a different symptom of the same disease.
There is a useful ordering underneath this. A container image pull is a stack of preconditions, and each one can only fail once the one below it succeeded:
6. manifest / layers available
5. repository permissions
4. registry authentication
3. TLS handshake
2. TCP reachability
1. DNS resolution <- failed hereMy instinct was to start at 4, because “pull failed” pattern-matches to “credentials”. Starting at 1 costs ten seconds and rules out five layers.
Case 2: the port was in use, by nothing I could see
Different day. A container would not start because its host port was taken.
docker ps showed nothing on that port. No container was publishing it. I
restarted the daemon, which achieved what restarting the daemon usually
achieves.
Docker was right, and Docker was not the right place to look. Port bindings are a property of the host’s network stack, not of Docker’s bookkeeping. Any process on the machine can hold one:
sudo ss -ltnp | grep ':5001'
sudo lsof -iTCP:5001 -sTCP:LISTEN -n -PA monitoring agent, installed as a host service and completely outside Docker,
was listening on 127.0.0.1:5001. It had been there for months. Nothing had
needed that port until this deployment.
ps -fp <pid> # what is this process?
systemctl list-units --type=service | grep -i <name> # what installed it?Then the same principle as before, from the inside out: the kernel owns the socket table, Docker only asks for a binding. Ask the kernel.
The habit
When an error has multiple clauses, find the innermost one first. Some practical markers of where the real information sits:
A syscall.Errno, an errno value, or a bare number in a detail field is
the kernel speaking, and the kernel does not speculate. A port number tells you
which protocol failed regardless of what the application layer claims it was
doing. connection refused and timed out are different diagnoses: refused
means something answered, timed out usually means a firewall or a dead host.
And a hostname in an error means name resolution was at least attempted, which
narrows the timeline of what already worked.
The general rule that came out of both of these:
The component that reports an error is rarely the component that has the problem. It is the first one that could not continue.
That gap is where the hours go. Read the message backwards, find the smallest claim it makes, and test that claim on its own. If it fails, everything above it is noise, including the part in bold at the top.