Debugging Kubernetes Pods: CrashLoopBackOff, OOMKilled, ImagePullBackOff, Pending and Probe Failures

By 0xmote · Aug 8, 2026 · 15 min read · Updated Sep 24, 2026 Intermediate tutorial

Key takeaways

Most broken pods can be diagnosed from three places: the container's Last State and exit code, the Events at the bottom of kubectl describe, and the logs of the previous container instance. This guide shows how to read each for CrashLoopBackOff, OOMKilled, ImagePullBackOff, Pending and probe-induced restarts, and which fixes actually address the cause.

The Three Places the Answer Usually Is

When a pod misbehaves, resist the urge to start changing YAML. Nearly every pod problem leaves evidence in one of three places:

The container’s state and exit code, shown under State and Last State in kubectl describe pod.
The Events section at the bottom of kubectl describe pod, written by the scheduler and the kubelet.
The logs of the previous container instance, via kubectl logs --previous.

kubectl get pods -o wide                     # status, restarts, node
kubectl describe pod <pod>                   # state, last state, events
kubectl logs <pod> --previous                # output of the crashed instance
kubectl logs <pod> -c <container> --previous # multi-container pods
kubectl get events --sort-by=.lastTimestamp  # namespace-wide timeline

Two things catch people out early. First, plain kubectl logs shows the current instance, which during a crash loop is often a fresh container that has not logged anything yet; the error you want is almost always in --previous. Second, events are short-lived: the API server keeps them for one hour by default, so for an incident that happened overnight the events are likely gone and you have to rely on your log and metrics pipeline.

Also be careful with kubectl get pods --field-selector=status.phase!=Running as a “find broken pods” command. A pod in CrashLoopBackOff usually still has phase Running (its restart policy keeps it alive), so this selector misses exactly the pods you are looking for. Look at the STATUS and RESTARTS columns instead.

CrashLoopBackOff

CrashLoopBackOff is not an error in itself. It means the container keeps exiting and the kubelet is waiting before the next restart. The wait grows exponentially (10s, 20s, 40s, and so on) up to a cap of five minutes, and resets once a container has run successfully for ten minutes. That is why a crashing pod seems to “hang” for minutes between attempts: the kubelet is backing off, not stuck.

The cause is whatever made the process exit. Start with describe:

    State:          Waiting
      Reason:       CrashLoopBackOff
    Last State:     Terminated
      Reason:       Error
      Exit Code:    1
      Started:      ...
      Finished:     ...

Reading the exit code

Exit code	What it means	Where to look
0	Process exited successfully	A long-running Deployment whose main process finishes (a script, a command that daemonizes into the background) is restarted forever
1 (or other small numbers)	Application error	`logs --previous`: stack trace, config validation failure, failed DB connection
126	Command found but not executable	Missing execute bit, wrong entrypoint file
127	Command not found	Typo in `command`/`args`, binary not in the image, wrong `PATH`
137	SIGKILL (128 + 9)	`Reason: OOMKilled` means memory limit; otherwise killed after grace period (liveness failure, rollout)
139	SIGSEGV (128 + 11)	Native crash; often a native library built for a different libc
143	SIGTERM (128 + 15)	Process stopped because it was asked to: liveness restart, rollout, node drain

The exit-code-0 case surprises people: Kubernetes Deployments expect the main process to run forever, so a container whose entrypoint starts a server in the background and returns will “succeed” and be restarted endlessly. For run-to-completion work, use a Job.

A crash with no logs at all often means the process never started. Typical messages in logs --previous or the events:

exec /app/server: exec format error

That is an image built for the wrong CPU architecture, such as an arm64 image built on an Apple Silicon laptop running on amd64 nodes. Build multi-platform images (docker buildx build --platform linux/amd64,linux/arm64) or build in CI on the target architecture.

Missing configuration

A very common exit-1 cause is configuration that exists in one environment and not another. If a referenced Secret or ConfigMap key is missing, the container does not even start, and describe shows a state reason like CreateContainerConfigError with an event such as Error: secret "app-secrets" not found. If the variable is simply absent from the spec, the container starts and the app fails its own config validation, which shows up in logs --previous.

kubectl get pod <pod> -o jsonpath='{.spec.containers[0].env}'
kubectl get secret app-secrets -o jsonpath='{.data}'   # keys present? (values are base64)

OOMKilled (Exit Code 137)

    Last State:     Terminated
      Reason:       OOMKilled
      Exit Code:    137

This means the container’s memory usage reached its limits.memory and the kernel OOM killer terminated a process in the container’s cgroup. It is not graceful: no shutdown hooks, no final log line, which is why logs --previous often just stops mid-stream.

Distinguish it from eviction, which looks different and has a different fix. When a node itself runs low on memory, the kubelet evicts pods: they show status Evicted with a message like The node was low on resource: memory. Eviction targets pods using more than they requested, so the fix there is setting realistic requests, not raising limits.

resources:
  requests:
    memory: "384Mi"   # what the pod normally needs; used for scheduling and eviction ranking
  limits:
    memory: "512Mi"   # hard ceiling; exceeding it gets the container OOM-killed

Before simply raising the limit, check what the memory is:

Runtime heap settings. Managed runtimes size their heap from their own configuration, and the heap is not the whole process: thread stacks, buffers and native memory come on top. A JVM with -Xmx equal to the container limit, or a Node.js process with --max-old-space-size set close to the limit, will be OOM-killed before the runtime ever throws its own out-of-memory error. Leave headroom (for the JVM, -XX:MaxRAMPercentage sizes the heap relative to the container limit).
A leak. If memory climbs steadily until the kill, raising the limit only lengthens the interval between restarts. Take a heap snapshot or profile before the limit is reached.
A spike. Loading an entire file or result set into memory for one request can push a normally small pod over its limit.

kubectl top pod <pod> --containers shows current usage, but it requires metrics-server; without it you get error: Metrics API not available. Current usage right after a restart is also not very informative. Memory graphs from your monitoring stack over the hours before the kill tell you whether it is a leak or a spike.

The pattern I have seen most often here is the runtime heap sized to exactly the container limit. The app runs fine under light load, then gets OOM-killed under heavier traffic without ever logging an out-of-memory error, because the kernel killed it before the runtime’s own limit was reached. Everyone looks for a leak in the application logs, and the logs have nothing because the process was killed rather than failing. Once you know that OOMKilled means “the cgroup limit, not the heap”, the fix of leaving headroom between heap and limit is straightforward.

ImagePullBackOff and ErrImagePull

ErrImagePull is the failed attempt; ImagePullBackOff is the kubelet waiting before retrying, with the same exponential backoff as crash loops. The event message tells you which problem you have:

Event message contains	Cause
`not found` / `manifest unknown`	Tag or image name does not exist in that registry
`pull access denied` / `unauthorized` / `authentication required`	Private image without working credentials
`toomanyrequests` / `429 Too Many Requests`	Registry rate limit (Docker Hub anonymous pulls are rate-limited)
`no match for platform in manifest`	Image not built for the node’s architecture
`i/o timeout` / `dial tcp`	Node cannot reach the registry (egress, proxy, DNS)

For private registries:

kubectl create secret docker-registry regcred \
  --docker-server=registry.example.com \
  --docker-username=<user> \
  --docker-password=<token> \
  -n <namespace>

spec:
  imagePullSecrets:
    - name: regcred

The Secret must be in the same namespace as the pod; a secret created in default does nothing for a pod in prod. You can also attach imagePullSecrets to the namespace’s ServiceAccount so every pod gets it automatically.

A related trap is the :latest tag. With latest, the default imagePullPolicy is Always, so every pod start depends on the registry being reachable and not rate-limiting you, and two pods of the same Deployment can run different code if the tag moved between their starts. Use immutable tags (a version or commit SHA) or image digests.

Pending

A Pending pod has not been assigned to a node, or is waiting on volumes. The scheduler explains why in the Events:

Warning  FailedScheduling  default-scheduler  0/3 nodes are available: 1 node(s) had untolerated taint {node-role.kubernetes.io/control-plane: }, 2 Insufficient memory. preemption: 0/3 nodes are available: 1 Preemption is not helpful for scheduling, 2 No preemption victims found for incoming pod.

Read it as a tally per node: here one node is excluded by a taint and two do not have enough unrequested memory.

Insufficient cpu / memory

The scheduler works with requests, not actual usage. A node running at low real utilisation can still be “full” if other pods requested more than they use. Check allocated requests per node:

kubectl describe node <node> | grep -A 8 "Allocated resources"

Fixes, in the order I would try them: lower requests that are clearly inflated, check whether a namespace LimitRange is injecting large default requests, then add nodes or let the cluster autoscaler do so. Note also that a single pod’s request must fit on one node; a pod requesting more memory than any node’s allocatable capacity will never schedule, no matter how many nodes you add.

Taints, node selectors and affinity

kubectl get nodes --show-labels
kubectl describe node <node> | grep -i taints

A pod with a nodeSelector or required node affinity only goes to nodes with matching labels. A tainted node (GPU pools, dedicated tenants) only accepts pods that tolerate the taint:

tolerations:
  - key: "dedicated"
    operator: "Equal"
    value: "gpu"
    effect: "NoSchedule"

A toleration allows scheduling onto the tainted node but does not require it. To pin workloads to that pool you need both the toleration and a node selector or affinity.

Volumes

kubectl get pvc
kubectl describe pvc <claim>

pod has unbound immediate PersistentVolumeClaims means the claim is not bound: wrong or missing StorageClass, no provisioner, or no matching PersistentVolume. volume node affinity conflict usually means the volume exists in a different availability zone than every node that could run the pod; zonal block storage cannot attach across zones.

Probe Failures: Readiness vs Liveness vs Startup

Probes are a frequent cause of restarts that look like application crashes.

Probe	On failure	Use for
`readinessProbe`	Pod removed from Service endpoints, no restart	”Can I take traffic right now?”
`livenessProbe`	Container is killed and restarted	”Am I stuck beyond recovery?”
`startupProbe`	Container restarted if it never succeeds; liveness/readiness wait until it passes	Slow startup

When liveness kills a container, the events say so:

Warning  Unhealthy  kubelet  Liveness probe failed: Get "http://10.0.1.23:8080/healthz": dial tcp 10.0.1.23:8080: connect: connection refused
Normal   Killing    kubelet  Container app failed liveness probe, will be restarted

and Last State typically shows exit code 143 (the app exited on SIGTERM) or 137 (it did not exit within the grace period).

The classic misconfiguration is a liveness probe on an app that takes longer to start than initialDelaySeconds + failureThreshold * periodSeconds. The container is killed while still starting, restarts, is killed again, and ends up in CrashLoopBackOff with nothing wrong in the application. Rather than inflating initialDelaySeconds (which also delays detection of real hangs later), add a startup probe:

startupProbe:
  httpGet: { path: /healthz, port: 8080 }
  periodSeconds: 5
  failureThreshold: 30       # allows up to 150s to start
livenessProbe:
  httpGet: { path: /healthz, port: 8080 }
  periodSeconds: 10
  failureThreshold: 3
  timeoutSeconds: 2
readinessProbe:
  httpGet: { path: /ready, port: 8080 }
  periodSeconds: 5
  failureThreshold: 3

The second misconfiguration is a liveness endpoint that checks dependencies. If /healthz returns 500 when the database is slow, a database hiccup makes the kubelet restart every replica at the same time, which turns a partial degradation into a full outage and adds a thundering herd of reconnects. Liveness should check only that the process itself is responsive; put dependency checks, if anywhere, in readiness.

Probe restarts are the failure I would check first whenever a pod restarts with no stack trace in logs --previous. I have seen slow-starting services, often ones that warm caches or run migrations at boot, restart in a loop right after a deploy while the same image runs perfectly under docker run locally. The logs look like a clean startup that is interrupted partway through, and the only evidence is the Killing ... failed liveness probe event. Adding a startup probe is almost always the fix.

To test the probe endpoint from inside the pod (if the image has a shell and curl):

kubectl exec -it <pod> -- curl -sv http://localhost:8080/healthz

Running and Ready, but Traffic Fails

When pods look healthy but a Service does not work, check whether the Service has any endpoints:

kubectl get endpointslices -l kubernetes.io/service-name=<service>
kubectl get service <service> -o jsonpath='{.spec.selector}'
kubectl get pods -l app=<value-from-selector>

An empty endpoint list almost always means the Service selector does not match the pod labels (a typo, or labels changed in the Deployment template but not the Service), or the pods are not Ready. Also check that the Service’s targetPort is the port the container actually listens on.

For DNS and connectivity, test from a pod in the same namespace:

kubectl run net-debug --rm -it --image=nicolaka/netshoot -- bash
# inside:
nslookup my-service.my-namespace.svc.cluster.local
curl -sv http://my-service.my-namespace:8080/healthz

If DNS works and the connection times out, look at NetworkPolicies (kubectl get networkpolicy -n <namespace>). Once any policy selects a pod for ingress, all ingress not explicitly allowed is denied.

Init Containers

A pod stuck at Init:0/1 or Init:CrashLoopBackOff is waiting on or failing in an init container. Main containers do not start until every init container has exited successfully.

kubectl logs <pod> -c <init-container-name>

A typical pattern is waiting for a dependency:

initContainers:
  - name: wait-for-db
    image: busybox:1.36
    command: ['sh', '-c', 'until nc -z postgres 5432; do echo waiting for db; sleep 2; done']

Be aware that this only delays startup; the application still needs to handle the database going away later. An init container that waits forever on a dependency that never comes up will leave the pod in Init indefinitely, so pair it with alerting.

Containers Without a Shell

Minimal images (distroless, scratch) have no shell, so kubectl exec -it <pod> -- sh fails with something like exec: "sh": executable file not found in $PATH. Use an ephemeral debug container instead:

# Attach a busybox container that shares the target container's process namespace
kubectl debug -it <pod> --image=busybox:1.36 --target=<container>

# Or start a copy of the pod with a different command, to poke at a crashing container
kubectl debug <pod> -it --copy-to=<pod>-debug --container=<container> -- sh

With --target, you can see the target’s processes and reach its filesystem through /proc/<pid>/root. The copy approach is useful for crash loops: the copy runs your shell instead of the crashing entrypoint, with the same image, env and volumes.

Stuck in Terminating

A pod stuck in Terminating is usually waiting on one of: a process that ignores SIGTERM until terminationGracePeriodSeconds expires, a finalizer that a controller has not removed, or a node that is unreachable so the kubelet cannot confirm the container stopped.

kubectl get pod <pod> -o jsonpath='{.metadata.finalizers}'
kubectl get node <node>          # NotReady?

kubectl delete pod <pod> --grace-period=0 --force removes the object from the API immediately, but it does not guarantee the container has stopped on the node. For StatefulSet pods this can lead to two instances with the same identity running at once, so only force-delete when you know the node is really gone.

Status Quick Reference

STATUS column	Meaning	First command
`Pending`	Not scheduled or waiting for volumes	`describe pod` → FailedScheduling event
`ContainerCreating` (long)	Volume mount, image pull or network setup in progress or failing	`describe pod` → events
`CreateContainerConfigError`	Referenced Secret/ConfigMap/key missing	`describe pod` → event names the missing object
`Init:N/M`, `Init:CrashLoopBackOff`	Init container waiting or failing	`logs -c <init>`
`ErrImagePull`, `ImagePullBackOff`	Image pull failing	Event message (not found / denied / 429)
`CrashLoopBackOff`	Container keeps exiting	`logs --previous`, exit code in Last State
`OOMKilled`	Memory limit exceeded	Last State reason, memory graphs
`Evicted`	Node under resource pressure	Pod status message, node conditions
`Running` with `0/1` READY	Readiness failing	`describe pod` → Unhealthy events
`Terminating` (long)	Grace period, finalizer or lost node	Finalizers, node status