How Kubernetes Actually Works: Scheduler Scoring, etcd Raft, Reconciliation Loops and kube-proxy
Key takeaways
Beyond everyday YAML, this guide explains how Kubernetes actually works: how Pods are scheduled, how Services are implemented on the node, how etcd stores cluster state under Raft, how controllers reconcile desired state, and which production patterns matter in real clusters.
Most Kubernetes material stops at YAML: write a Deployment, expose it with a Service, apply. That is enough until a Pod sits in Pending, a Service silently drops traffic, or the control plane slows down. This article explains the machinery underneath: how the scheduler chooses a node, how etcd stores and replicates cluster state, how controllers converge on desired state, and how kube-proxy turns a Service IP into a Pod IP.
Architecture at a Glance
Kubernetes is built from a declarative API and control loops. You record desired state through the API (for example with kubectl apply), and controllers keep driving the live cluster toward it.
The control plane has a handful of components. kube-apiserver is the only component that talks to etcd; every request goes through authentication, authorization, and admission before it is persisted, and every other component reads and watches the cluster through the API server. etcd is the replicated key-value store that holds all API objects. kube-scheduler assigns unscheduled Pods to nodes. kube-controller-manager runs the built-in controllers (Deployment, ReplicaSet, Node, Job, EndpointSlice, and many more). On cloud platforms, cloud-controller-manager handles load balancers, routes, and node lifecycle against the provider’s API.
Each worker node runs kubelet, which watches for Pods assigned to its node and starts them through the Container Runtime Interface (CRI), typically containerd or CRI-O. kube-proxy programs node-level forwarding rules for Services, although some CNI plugins (Cilium, for example) can replace it with their own eBPF data plane. The CNI plugin provides Pod networking, and CSI drivers provide storage.
Core Objects
A Pod is the smallest deployable unit: one or more containers that share a network namespace (one IP, one port space) and can share volumes. Pods are disposable and are not recreated if their node dies, so in practice you create them through a controller such as a Deployment, StatefulSet, DaemonSet, or Job.
A Deployment declares the desired number of replicas, the Pod template, and the update strategy. It does not manage Pods directly; it manages one ReplicaSet per template revision, and each ReplicaSet keeps its Pod count at the target.
A Service gives a set of Pods, selected by labels, a stable virtual IP (ClusterIP) and a DNS name. The EndpointSlice controller tracks which matching Pods are ready and publishes their IPs and ports in EndpointSlice objects, which kube-proxy consumes.
An Ingress routes external HTTP(S) traffic to Services by host and path, but does nothing on its own: an Ingress controller has to implement it. The widely used community ingress-nginx controller has been retired by the Kubernetes project, and the Gateway API is the actively developed successor for new setups.
ConfigMaps and Secrets inject configuration into Pods. Secrets are only base64-encoded and are stored in etcd in plaintext unless you configure encryption at rest on the API server, so RBAC on Secrets and etcd access control matter as much as the object type.
Pod Scheduling
kube-scheduler watches for Pods with no spec.nodeName and runs each one through the scheduling framework. The main phases are filter, score, and bind, with extension points such as reserve and permit in between.
Filtering
Filtering removes nodes that cannot run the Pod. The default plugins check whether the Pod’s resource requests fit in the node’s allocatable CPU and memory minus the requests of Pods already there (actual usage is not considered), whether a requested host port is free, whether nodeSelector and required node affinity match, whether required inter-Pod affinity and anti-affinity rules are satisfied, whether the Pod tolerates the node’s taints, and whether the Pod’s volumes can be attached in the node’s zone. If no node passes, the Pod stays Pending and the scheduler records an event such as 0/12 nodes are available: 3 Insufficient memory, 9 node(s) had untolerated taint ..., which usually names the exact reason.
In large clusters the scheduler does not necessarily evaluate every node. Once it has found enough feasible nodes (controlled by percentageOfNodesToScore, which defaults to an adaptive value), it stops filtering and moves on to scoring.
Scoring
Each scoring plugin gives every feasible node a score from 0 to 100, and the scheduler sums the weighted scores. Default plugins include NodeResourcesFit (by default preferring nodes with more free capacity), NodeResourcesBalancedAllocation (preferring nodes where CPU and memory usage stay balanced), ImageLocality (preferring nodes that already have the container images), preferred node affinity and inter-Pod affinity, TaintToleration (preferring nodes with fewer PreferNoSchedule taints), and PodTopologySpread. The node with the highest total wins; ties are broken randomly.
Binding
The scheduler then writes a Binding for the Pod, which sets spec.nodeName. Nothing more happens in the scheduler: the kubelet on that node sees through its watch that a Pod has been assigned to it, pulls the images, sets up volumes and networking, and starts the containers.
Operational notes
Because filtering is based on requests rather than usage, Pods without resource requests look free to the scheduler and can pile onto one node until it runs out of memory. Setting requests on every production container is what makes scheduling and capacity planning meaningful. When the defaults are not enough, for example for GPU packing or bin-packing nodes to save cost, scheduler profiles let you change plugin weights or run a second profile for a subset of Pods.
etcd Consensus and Data Model
Raft
etcd replicates its data with the Raft consensus algorithm. One member is the leader; every write is appended to the leader’s log and is committed only once a majority of members have stored it. Clusters run an odd number of members, usually three or five, because a cluster of n members tolerates the loss of (n-1)/2 of them, and adding a fourth member to three raises the quorum without adding fault tolerance.
If a majority is lost, etcd cannot commit writes, so the API server can no longer create, update, or delete objects. The API server also reads from etcd with quorum (linearizable) reads by default, so many reads fail as well; only requests that can be served from the API server’s watch cache keep working. Workloads already running on nodes keep running, but nothing in the cluster can change. This is why etcd members are placed in separate failure domains.
Keys and objects
The API server stores each object under a key of the form /registry/<resource>/<namespace>/<name>, for example /registry/pods/default/web-7d4b9c-abcde, with cluster-scoped objects omitting the namespace. Built-in types are stored as protobuf, custom resources as JSON. etcd keeps multiple revisions of each key (MVCC); the API server’s resourceVersion is derived from etcd’s revision, and the API server periodically compacts old revisions so the database does not grow without bound.
Watches and the reconcile pattern
Controllers and the scheduler do not poll. They open watches through the API server, which itself watches etcd and fans changes out from a watch cache. Every change to an object is delivered to the components interested in it, and that event stream is what makes the reconciliation model below work.
Operations and security
Take regular etcd snapshots and practice restoring them. Nodes cannot reconstruct the cluster’s desired state on their own, so without a snapshot, losing etcd means rebuilding every object from your manifests. etcd should only be reachable by the API server over mutual TLS, because read access to etcd is read access to every Secret. etcd has a backend size quota (2 GiB by default); a cluster that creates many objects or large ones, for example thousands of ConfigMaps or Helm release history, can hit it and switch etcd into an alarm state that rejects writes.
Controller Reconciliation Loops
Kubernetes self-healing is the work of controllers that keep comparing observed state with desired state and act on the difference.
Most controllers follow the same structure. A shared informer lists and watches the object types the controller cares about and keeps a local cache, so reads do not hit the API server. When an object changes, its key (namespace/name) is added to a work queue; the queue deduplicates keys, so ten rapid changes to one object result in a single reconcile. Worker goroutines take keys off the queue and run a reconcile function that reads the current state from the cache, decides what is missing or extra, and creates, deletes, or patches objects through the API server. Failures are requeued with exponential backoff.
The important property is that reconcile works from the current state, not from the event that triggered it. If the controller restarts or misses an event, the next reconcile still converges, which is why controllers are described as level-triggered rather than edge-triggered.
The Deployment controller manages ReplicaSets, one per Pod template revision. During a rolling update it scales the new ReplicaSet up and the old one down step by step, within the limits of maxSurge (how many Pods above the desired count may exist) and maxUnavailable (how many may be missing). The ReplicaSet controller in turn creates or deletes Pods to match its replica count, and marks them with owner references so garbage collection can clean them up.
This structure explains several debugging situations. A Deployment that converges slowly may be waiting on an admission webhook that times out on every Pod creation, or on Pods that never become ready so the rollout cannot proceed. Pods that are created and immediately deleted, or that two controllers fight over, usually come from overlapping label selectors; kubectl get pod -o yaml shows ownerReferences, which tell you which controller owns a Pod.
Service Networking and kube-proxy Modes
A Service gets a virtual IP from the cluster’s Service CIDR, and CoreDNS resolves <service>.<namespace>.svc.cluster.local to that IP. No network interface actually has the ClusterIP. Instead, kube-proxy on each node watches Services and EndpointSlices and programs the node’s kernel so that packets sent to the ClusterIP and port are rewritten to the IP and port of one ready backend Pod.
iptables mode
In iptables mode, the default on Linux, kube-proxy writes NAT rules. A packet to a Service IP jumps from the KUBE-SERVICES chain to a per-Service chain, which picks one endpoint chain at random using the statistic match with precomputed probabilities, and the endpoint chain DNATs the packet to the Pod IP. Connection tracking keeps later packets of the same connection on the same Pod.
The cost shows up at scale. Matching a Service walks the KUBE-SERVICES chain, so lookup time grows with the number of Services, and every change has kube-proxy restore large rule sets, which can take seconds in clusters with tens of thousands of endpoints. Because the backend is chosen per connection, long-lived connections such as HTTP/2 or gRPC stay pinned to whichever Pod they first reached; spreading those requests needs client-side or proxy-level load balancing.
nftables mode
The nftables mode, GA in Kubernetes 1.33, uses nftables verdict maps so a Service lookup is a single map lookup regardless of how many Services exist, and it updates rules incrementally instead of rewriting them. It requires a reasonably recent kernel; upstream now recommends it for new large clusters where supported.
IPVS mode
IPVS mode uses the kernel’s IP Virtual Server load balancer, which stores Services in hash tables and offers schedulers such as round robin and least connections. It was the traditional answer to iptables scaling problems, but it still depends on iptables for some features and has a history of behavior differences from the other modes, and upstream now directs users who need performance toward nftables instead.
The original userspace mode, where kube-proxy itself proxied every connection, was removed in Kubernetes 1.26.
NodePort and LoadBalancer
A NodePort Service opens the same port (from the 30000-32767 range by default) on every node and forwards it to the Service, so firewall rules and security groups have to account for it. A LoadBalancer Service asks the cloud controller to provision an external load balancer that sends traffic to the node ports or directly to Pods, depending on the provider. With the default externalTrafficPolicy: Cluster, traffic can be forwarded to a Pod on another node and is SNATed on the way, so the Pod sees a node IP instead of the client IP; externalTrafficPolicy: Local preserves the client IP but only sends traffic to nodes that run a ready Pod.
Troubleshooting
When traffic to a Service fails, the chain to check is short: the Service selector must match the Pod labels, the EndpointSlices must list ready addresses, and the Service targetPort must be the port the container actually listens on. Empty endpoints almost always mean a selector mismatch or failing readiness probes.
NetworkPolicies need a CNI plugin that enforces them; on a CNI that does not, they are silently ignored. Until a Pod is selected by some policy, all traffic to and from it is allowed. Once an egress policy selects a Pod, everything not explicitly allowed is blocked, including DNS, so the first symptom of a new egress policy is often name resolution failing. Allow UDP and TCP port 53 to the cluster DNS Pods explicitly.
Production Patterns
Run at least two replicas of anything that must stay up, and add a PodDisruptionBudget so that node drains during upgrades evict Pods only while enough replicas remain available. Without a PDB, a drain can evict all replicas of a Deployment at once if they happen to share a node; topology spread constraints reduce that risk by spreading replicas across nodes and zones in the first place.
For security, grant RBAC permissions per namespace with the least privilege needed and avoid binding broad cluster roles to workloads. Admission policies, using the built-in ValidatingAdmissionPolicy or tools such as OPA Gatekeeper and Kyverno, enforce rules like “no privileged containers” or “images only from our registry” at the API level instead of relying on review. NetworkPolicies restrict east-west traffic between namespaces. For Secrets, enable encryption at rest, and consider external secret stores (through the Secrets Store CSI driver or External Secrets Operator) when you need rotation and audit outside the cluster.
ResourceQuota caps the total resources a namespace can request, and LimitRange sets default requests and limits for containers that do not specify them, which closes the “Pods without requests” gap described above. The Horizontal Pod Autoscaler changes the number of Pods based on metrics, while the cluster autoscaler (or Karpenter) adds nodes when Pods are Pending for lack of capacity and removes underused nodes. They solve different problems and are usually used together: HPA creates Pods, the node autoscaler finds room for them.
For observability, kube-state-metrics exposes the state of API objects (desired vs. available replicas, Pod phases, restarts), and the kubelet’s cAdvisor endpoint exposes container resource usage. Alerts worth having from the start are API server latency and error rates, etcd leader changes and database size, Pods stuck in Pending, OOMKilled containers, and CrashLoopBackOff.
GitOps tools such as Argo CD and Flux keep manifests in Git and continuously reconcile the cluster against them, which gives you an audit trail and rollbacks through Git history. They also reveal drift: a change someone made with kubectl edit shows up as out of sync and is reverted on the next sync unless it is committed.
kubectl Quick Reference
kubectl get pods -A -o wide
kubectl describe pod POD -n NAMESPACE
kubectl get endpointslices -n NAMESPACE -l kubernetes.io/service-name=SERVICE
kubectl get events -n NAMESPACE --sort-by=.lastTimestamp
kubectl get networkpolicy -A
kubectl explain pod.spec.affinity