Kubernetes is remarkably good at self-healing: a container crashes, it restarts; a node dies, work reschedules elsewhere. That resilience is the point — but it's also a monitoring trap. K8s will quietly paper over problems right up until it can't, and by then the issue is widespread. Monitoring Kubernetes is about seeing through the magic.
The layers to watch
Kubernetes has levels, and you need visibility at each:
| Layer | What lives here | Watch for |
|---|---|---|
| Pods | Your running containers | Restarts, crashes, OOM kills, pending pods |
| Nodes | The machines pods run on | CPU/memory pressure, disk, "NotReady" |
| Cluster | The control plane | API health, scheduling, etcd |
| Workloads | Deployments/replicas | Desired vs available replicas |
The signals that matter most
Pod restarts and CrashLoopBackOff
A high restart count is the classic "Kubernetes is hiding it" signal. A pod crashing and restarting every minute (the dreaded CrashLoopBackOff) looks "running-ish" from a distance but is effectively broken. Watch restart counts — a climbing number is an alarm.
Pending pods and resource pressure
Pods stuck Pending usually mean the cluster can't schedule them — out of CPU/memory, or no node fits. That's a capacity warning before it becomes an outage. Watch node CPU/memory pressure and pods that can't be placed.
Desired vs available replicas
A deployment wants 5 replicas but only 3 are available? You're running degraded. The gap between desired and available is one of the highest-signal things to alert on.
Liveness and readiness probes
Kubernetes uses two health checks you should understand and tune:
- Liveness probe — "is this container alive?" Fails → K8s restarts it.
- Readiness probe — "is it ready for traffic?" Fails → K8s stops sending it requests.
Misconfigured probes cause their own outages (restart loops, or traffic to not-ready pods), so monitor that they're behaving.
Watch symptoms, not just internals
Like with microservices, a cluster has thousands of metrics. Alerting on all of them is a one-way ticket to alert fatigue.
Alert on impact, not on every restart. A single pod restarting is often Kubernetes doing its job — fine. Many restarts, replicas unavailable, or pods stuck pending means the self-healing is failing — that's worth a page. Tune alerts to "the magic isn't working anymore."
And still — check from outside
Kubernetes can report everything green internally while users can't reach the app (an ingress misconfig, a DNS problem, a bad load balancer). So keep external uptime checks on your public endpoints, from multiple locations. They're the reality check that internal cluster metrics can't give you.
The bottom line
| In one line | |
|---|---|
| The trap | K8s self-heals so well it can mask building problems. |
| Watch | Pod restarts, pending pods, node pressure, desired vs available replicas. |
| Probes | Liveness restarts; readiness gates traffic — tune both. |
| Alert on | Self-healing failing, not every restart. |
| Anchor | External checks on public endpoints = ground truth. |
Kubernetes hides failures by design — which is great, until it isn't. Watch the signals that show the self-healing reaching its limits, alert on real impact, and always confirm from outside that users can actually get in.
Related: Monitoring microservices, uptime monitoring 101.