Kubernetes basics:
what happens after kubectl apply

A Kubernetes cluster is a control plane, which stores the cluster’s state and acts on it, plus worker nodes, which run your containers. A node is any machine running a kubelet, the agent chapter 04 follows. The unit Kubernetes places on a node is a Pod: one or more containers that share one IP address.

This cluster comes from kind (Kubernetes in Docker) v0.33.0, which built three nodes as three Docker containers in 22.8 seconds: kind-control-plane, kind-worker and kind-worker2, running Kubernetes v1.37.0, released on 26 August 2026. In chapter 07, stopping one of those containers is how a node loses power.

The control plane carries a taint, node-role.kubernetes.io/control-plane:NoSchedule. A taint repels every Pod that doesn’t explicitly tolerate it, so none of your Pods will run there. Chapter 03 shows the scheduler applying that rule.

The kubelet on kind-control-plane reads four YAML files from /etc/kubernetes/manifests and runs each one as a static Pod. A static Pod needs no API server, and that is the point: at boot there isn’t one yet. The kubelet starts the API server itself, from a file.

The four Pods are the control plane. kube-apiserver is the only door to cluster state. etcd is the key-value store that holds that state. kube-scheduler picks a node for each new Pod. kube-controller-manager runs the control loops that turn what you asked for into what exists.

Once the API server is up, it lists each static Pod as a read-only mirror, so kubectl get pods -n kube-system shows all four. A cloud cluster adds a fifth component, cloud-controller-manager, which talks to the provider’s API. kind runs on a laptop, so it has none.

Every node, the control plane included, runs the same three programs. The kubelet starts the Pods bound to its node and reports on them. kube-proxy writes the packet rules behind Services, which chapter 05 reads line by line. containerd 2.3.4 runs the containers, and the kubelet drives it through the CRI, a gRPC API.

kube-proxy runs as a Pod, one per node: kube-proxy-rkr7w, kube-proxy-mscwz and kube-proxy-ldszb. The kubelet and containerd are plain systemd services on the node’s own operating system. They have to be: something must already be running before the first Pod can start.

The other Pods are add-ons: they run as ordinary workloads and can be swapped. CoreDNS answers DNS queries for Service names and runs as a Deployment of 2 Pods. kindnet is the CNI plugin that gives every Pod an IP address, with one Pod per node. local-path-provisioner creates volumes on a node’s disk, from its own namespace.

The count: 4 static Pods, 3 kube-proxy, 3 kindnet, 2 CoreDNS and 1 provisioner make 13. Production clusters usually swap kindnet for Calico or Cilium, and the provisioner for their cloud’s storage driver. The rest stays the same.

A namespace is a name scope for objects and a boundary for permissions and quotas. A new cluster has four: default, kube-system for the cluster’s own 12 Pods, kube-public and kube-node-lease. kind adds a fifth, local-path-storage, for its provisioner.

kube-node-lease holds one Lease object per node. Each kubelet renews its Lease every 10 seconds, and that renewal is the node’s heartbeat. Chapter 07 stops one node and follows what the control plane does once the renewals stop.

Before your app exists, etcd already holds 408 keys under /registry in a 1.6 MB database, against a default quota of 2 GiB. 141 of them are cluster-wide RBAC roles and bindings. Nodes are stored under /registry/minions, the name early Kubernetes releases used for nodes.

kubectl is an HTTPS client. It reads the API server’s URL and your client certificate from a kubeconfig file, ~/.kube/config, then sends plain REST requests. kubectl apply -f web.yaml sent 5 of them.

Two GETs fetched the OpenAPI schema, so kubectl could check the manifest’s field names before sending anything. A GET for deployments/web returned 404 Not Found, which is how apply decides to create rather than patch. A GET for the namespace returned 200. Then one POST: 201 Created in 4 ms.

kubectl create namespace shop had run a moment earlier, one POST of its own. Those two are your first writes, so Σ writes to cluster state is now 2.

Every request passes the same checks, in order. Authentication: here a client certificate whose CN is kubernetes-admin and whose O is kubeadm:cluster-admins. Authorization: RBAC finds a ClusterRoleBinding that grants that group the cluster-admin role. Then admission, schema validation and, only after all of them, storage in etcd.

Admission is the one gate that can change the object. Mutating plugins run first and may edit it; validating plugins run next and may refuse it. The DefaultTolerationSeconds plugin, for one, adds two 300-second tolerations to every Pod created later in this tutorial, and chapter 07 depends on them.

A refusal ends the request with 401 Unauthorized, 403 Forbidden or 422 Unprocessable Entity, and nothing is stored. Kubernetes keeps no user database: a user is whatever name a trusted certificate or token carries.

The API server fills in every field you left out before it stores the object. web.yaml says nothing about how to update Pods, so the stored Deployment gets strategy RollingUpdate with maxSurge 25% and maxUnavailable 25%. The Pod template gains imagePullPolicy IfNotPresent, restartPolicy Always, dnsPolicy ClusterFirst and terminationGracePeriodSeconds 30.

It also adds what only the server can know: a uid, a creationTimestamp, managedFields recording which client set which field, and kubectl’s own last-applied-configuration annotation. kubectl get deploy web -o yaml returns 81 lines for a 33-line file.

The object is stored as protobuf, a compact binary encoding, under the etcd key /registry/deployments/shop/web: 3,069 bytes, starting with the four bytes k8s\0.

etcd keeps one revision counter for the whole cluster. Every write, by anyone, takes the next number, and each object’s resourceVersion is the revision of its own last write. Deployment web was stored at revision 845.

From revision 841, just before the apply, to 906, when all three Pods were Ready 5.1 seconds later, etcd took 65 writes: 35 for the shop app, 22 Events and 8 Lease renewals.

The number is also a lock. An update carries the resourceVersion its writer read, and if the object has changed since, the API server refuses it with 409 Conflict. The deployment-controller hit 2 conflicts in this run, read the object again and retried. Neither refused update counts as a write.

After the write, the API server calls no one. Each component instead keeps a watch open: a long-lived GET that streams every change to the resources it handles. kube-controller-manager watches Deployments, ReplicaSets and Pods; the scheduler watches Pods with no node; each kubelet watches Pods bound to its own node.

ReplicaSet web-67fddbbb7 was stored 5 ms after the Deployment, at 20:11:46.848, by a controller that had only read a watch event. Components never call each other, and only the API server reads or writes etcd. The Kubernetes docs treat access to etcd as equal to root on the whole cluster.

A controller reads the desired state from an object’s spec, counts what actually exists, and acts on the difference. A room thermostat works the same way: you set 21 °C, it measures the room, and it only switches the heater on when the two differ. Then the controller waits for the next watch event and runs the same comparison again.

kube-controller-manager runs 46 such loops in one process, for Deployments, ReplicaSets, StatefulSets, EndpointSlices, nodes and more. Each loop watches only the resources it handles. With several control-plane nodes every node runs a copy, but only the copy holding a Lease in kube-system is active. In this run it renewed that Lease 241 times, once every 2 seconds.

At 20:11:46.848, 5 ms after your Deployment was stored, the deployment-controller created ReplicaSet web-67fddbbb7. The suffix is a hash of the Pod template, and every Pod carries it again as the label pod-template-hash. Apply the same template to a fresh Deployment and you get the same name.

The replicaset-controller then saw replicas 3 against 0 matching Pods and created d9sc4, pwq2w and 6h5g4 between .851 and .857, 6 ms in all. Each child records an owner reference, a pointer to the object that created it, so deleting the Deployment also deletes its ReplicaSet and their Pods.

The split is deliberate. A Deployment manages ReplicaSets, one per version of the template, and only a ReplicaSet creates Pods. Chapter 07 depends on it: a rolling update is two ReplicaSets changing size at the same time.

A ReplicaSet keeps no list of its Pods. On every watch event it counts the Pods whose labels match its label selector, here app=web plus pod-template-hash, and compares that count with replicas. Labels are free-form key-value pairs, and you can change them on a running Pod.

After the measured run, the same cluster was used to test this. Relabelling web-d75cc9d86-dwv6l to app=debug made the replicaset-controller remove its owner reference 9 ms later and create a fourth Pod, zf8xk, 16 ms after the change. The relabelled Pod kept running, owned by nothing. Services, in chapter 05, find their Pods by label in the same way.

Every object has a spec, what you asked for, and a status, what exists, which controllers and kubelets write. Of this chapter’s 17 writes, 11 are status updates: 6 to the Deployment and 5 to the ReplicaSet, each recording a change in how many Pods exist or are ready. kubectl get reads these fields; it counts nothing itself.

The other 6 writes create objects: one ReplicaSet and three Pods from the apply, and two from the namespace alone. The root-ca-cert-publisher created ConfigMap kube-root-ca.crt and the service-account-controller created ServiceAccount default, both 4 ms after shop existed. Σ writes to cluster state is now 19.

Workload controllers differ in what they promise. A Deployment runs interchangeable replicas and replaces them in rolling updates. A StatefulSet gives each Pod a stable name and its own storage (chapter 06). A DaemonSet runs one Pod on every node. A Job runs Pods until a set number succeed, and a CronJob creates a new Job on a schedule.

This cluster ran two of these kinds before shop existed: DaemonSets for kube-proxy and kindnet, and Deployments for CoreDNS and local-path-provisioner. Join a fourth node and the daemon-set-controller creates a kube-proxy Pod and a kindnet Pod for it, without anyone applying anything.

A new Pod is a stored record with spec.nodeName: "" and status.phase: Pending. No container exists anywhere yet. No node has been told about the Pod, because no node has been chosen, and choosing one is the only job of kube-scheduler.

The scheduler watches for Pods with no node whose schedulerName is default-scheduler, the value the API server fills in when a manifest names none. It puts them in a queue and takes them one at a time, highest priority first. A Pod that fits nowhere goes back in the queue and stays Pending.

For each Pod the scheduler first filters out nodes that cannot run it. kind-control-plane fails at once: it carries the taint node-role.kubernetes.io/control-plane:NoSchedule, and the web Pods have no matching toleration. Other filters check free CPU and memory against requests, node affinity and host ports.

The two workers pass and go on to scoring. Plugins such as balanced resource use, image locality and topology spread give each node a score, and the highest wins. Below 100 nodes the scheduler scores every feasible node; on a 5,000-node cluster it stops searching once it has found feasible nodes for 10% of the cluster.

The scheduler adds up requests, never live usage. Each web Pod requests cpu: 100m, a tenth of a core, and memory: 64Mi. A node fits a Pod when its allocatable capacity, minus the requests of the Pods already placed there, still covers the new one.

Each kind node reports allocatable cpu 12, memory 12232528Ki (about 11.7 GiB) and pods 110. All three nodes are containers on one laptop VM with 12 CPUs, so the scheduler sees 36 CPUs that are really 12. It has no way to check: it trusts what each kubelet reports.

Limits are enforced on the node, not by the scheduler. Past 128 MiB the kernel kills the container, and the kubelet reports it as OOMKilled. Requests below limits give the Pod the QoS class Burstable, which decides its place in line when a node runs short of memory.

The scheduler’s output is a Binding object, POSTed to the Pod’s binding subresource. The API server applies it by setting spec.nodeName. Three Bindings landed in 4 ms, from 20:11:46.859 to .863: d9sc4 and 6h5g4 to kind-worker2, and pwq2w to kind-worker.

The scheduler never contacts a node. Each kubelet watches for Pods bound to its own node and learns about its new Pods from that watch, as chapter 04 shows. One Binding per Pod is the scheduler’s whole output, and these three take Σ writes to cluster state from 19 to 22.

The result was a 2/1 split: kind-worker2 got two replicas and kind-worker one. The scheduler’s built-in spreading is soft. Its default constraint allows nodes to differ by up to 3 Pods (maxSkew: 3 per hostname) with whenUnsatisfiable: ScheduleAnyway, so a 3/0 split would also have passed.

Forcing spread is opt-in: add topologySpreadConstraints with whenUnsatisfiable: DoNotSchedule, or Pod anti-affinity, to the template. To steer Pods toward nodes, use nodeSelector or node affinity. To keep them off a node, taint it, and give the Pods that may still run there a toleration, a matching exception in the Pod spec.

The kubelet on kind-worker2 keeps one watch open on Pods with the field selector spec.nodeName=kind-worker2. When the scheduler’s Binding for web-67fddbbb7-d9sc4 set that field at 20:11:46.859, the Pod appeared on that watch. The kubelet’s first status patch for it was stored 15 ms later.

The API server also limits what a kubelet may read. Its Node authorizer lets the kubelet on kind-worker2 read only the Pods bound to kind-worker2, and the Secrets, ConfigMaps and volumes those Pods name. A compromised node can’t list every Secret in the cluster.

The kubelet doesn’t start containers itself; it calls containerd over CRI. RunPodSandbox comes first: containerd starts the pause container, registry.k8s.io/pause:3.10, 268 kB, whose only job is to hold the Pod’s network namespace open. containerd then calls the CNI plugin, kindnet, which gives the sandbox 10.244.2.2 from this node’s range, 10.244.2.0/24.

Only then come PullImage, CreateContainer and StartContainer for nginx. Every container in a Pod joins the sandbox’s network namespace, so they share one IP and reach each other on localhost. On kind-worker2, crictl pods lists 4 sandboxes and crictl ps 4 containers: two web Pods, kube-proxy and kindnet.

nginx:1.30-alpine is 25,987,805 bytes. The kubelet on kind-worker pulled it in 3.239 s, and the one on kind-worker2 in 3.099 s. kind-worker2’s second Pod, web-67fddbbb7-6h5g4, asked for the same image at the same moment and waited for that download: its Pulled event reads 770 ms, or 3.869 s including the wait.

containerd caches images per node, so the next Pod on kind-worker2 starts without a download. For an image with a tag other than :latest, the default imagePullPolicy is IfNotPresent: pull only when the node lacks it. For :latest or no tag at all, it is Always.

The image format is OCI, an open standard for images and runtimes. Kubernetes removed its Docker Engine support, the dockershim, in v1.24 on 2022-05-03. Images built with docker build still run on containerd unchanged, because both follow OCI.

A started container puts its Pod in phase Running, but a Service sends it traffic only after its readiness probe passes. Here the kubelet runs GET / on port 80 every 5 s. Each Pod got 5 status patches, and the audit log shows what each one set: Initialized, sandbox ready, Running with an IP, Ready, and the CPU the container actually got.

Running to Ready took 292, 539 and 779 ms, because nginx answers at once; a slow application can sit Running and unready for much longer. The fifth patch reports 79m of CPU, not the 100m requested: the kernel stores the request as a coarse cgroup CPU weight, and reading that weight back rounds it down.

A liveness probe is different: when it fails, the kubelet restarts the container. Repeated crashes back off 10 s, 20 s, 40 s and so on, up to 5 minutes, shown as CrashLoopBackOff. A startup probe holds both off for slow starters. The kubelets’ 15 status patches bring Σ writes to 37.

Measured from the POST being stored at 20:11:46.843: the ReplicaSet at +5 ms, the three Pods at +8 to +14 ms, the Bindings at +16 to +20 ms, the first kubelet status patch at +31 ms. The control plane’s part was finished by then. The downloads took about 3 s, the last Pod was Ready at +5,115 ms, and kubectl rollout status returned at +5,179 ms.

Each stage also left an Event, a short record such as Scheduled, Pulling, Pulled, Created or Started: 22 in namespace shop for this apply. kubectl get events and kubectl describe pod read them, and the API server keeps them for 1 hour by default. Events aren’t in the Σ count: they report what happened rather than what exists.

Every Pod gets its own IP from its node’s slice of the Pod network, 10.244.0.0/16: 10.244.1.2 on kind-worker, 10.244.2.2 and 10.244.2.3 on kind-worker2. The network model is flat: any Pod can reach any other Pod’s IP directly, across nodes, without NAT. Making that true is the CNI plugin’s job.

None of those three Pods lasts eight minutes. The rollout in chapter 07 replaces all of them, and when redis-0 was recreated at 20:19:48 it was given 10.244.1.2, the address web Pod web-67fddbbb7-pwq2w held at 20:11. A client that had saved that Pod IP would now be talking to Redis.

kubectl apply -f web-svc.yaml created a Service named web: type ClusterIP, selector app=web, port 80. The API server gave it 10.96.192.87 from the Service range, 10.96.0.0/12, which is separate from the Pod range. It is your only write in this chapter, and it takes Σ writes to 38.

No interface holds that address and no process listens on it. Packets sent to it are rewritten on the sending node, before they leave, to a real Pod IP. On the reference cluster, ping 10.96.192.87 got no replies, because no rule rewrites ICMP, while HTTP to the same address returned 200 OK.

A Service object holds no addresses. The endpointslice-controller watches Services and Pods and writes an EndpointSlice, web-gbr96, listing each Pod behind web with its IP, node and readiness. Only Pods whose readiness probe passes are marked ready, and kube-proxy sends traffic only to those.

The endpoint-controller writes the same list a second time, into the older Endpoints object web. Endpoints has been deprecated since v1.33 and is still written in v1.37, so every address change in this run cost 2 writes: 24 to EndpointSlices and 19 to Endpoints. The two creates brought Σ writes to 40.

kube-proxy runs on every node, watches Services and EndpointSlices, and turns them into packet rules in the node’s kernel; it writes nothing to the API. In iptables mode, still the default in v1.37 (nftables went GA in v1.33, ipvs was deprecated in v1.35), it wrote 11 rules on kind-worker for shop/web.

One rule matches 10.96.192.87:80, one marks traffic from outside the Pod network for masquerading, and 3 chained rules pick a Pod. Each Pod then gets 2: a mark, and a DNAT that rewrites the destination address to the Pod’s IP. The first pick fires with probability 1/3, the second with 1/2 of what is left, and the third takes the rest.

CoreDNS, the cluster’s DNS server at 10.96.0.10, answers web.shop.svc.cluster.local with 10.96.192.87. The kubelet writes each Pod’s /etc/resolv.conf with that nameserver, three search suffixes and options ndots:5. The search list is why plain web works inside namespace shop, and web.shop from any other namespace.

ndots:5 means a name with fewer than 5 dots is tried with each search suffix before it is tried as written. A lookup of example.com, with 1 dot, sends CoreDNS three cluster names first, and each comes back NXDOMAIN. A trailing dot, example.com., marks the name as complete and skips the search list.

A ClusterIP is reachable only from inside the cluster. A NodePort Service also opens one port from 30000–32767 on every node’s IP. A LoadBalancer Service asks a cloud controller for an external load balancer in front of those ports; on plain kind nothing answers, so its external IP stays <pending>.

Routing HTTP by host name and path needs one more layer. An Ingress object routes through an Ingress controller. The Ingress API is frozen, and its successor is Gateway API, GA at v1.0 on 2023-10-31 and at v1.7.0 since 2026-10-29. ingress-nginx, the most widely used Ingress controller, was retired: best-effort maintenance ended in March 2026, and its repository is archived.

A ConfigMap is an API object that holds configuration as key-value pairs. redis-config has one key, redis.conf, whose value is two lines of Redis settings. The StatefulSet mounts the ConfigMap as a volume at /etc/redis, so each key becomes a file, and Redis reads /etc/redis/redis.conf as if it had been built into the image.

Values can also become environment variables, and the difference shows when the ConfigMap changes. The kubelet rewrites a mounted file in place on a later sync, while an environment variable keeps its old value until the Pod is recreated. A ConfigMap holds at most 1 MiB, because every object has to fit in one etcd request, and etcd’s limit is 1.5 MiB.

A Secret is a ConfigMap-shaped object meant for passwords, tokens and keys. kubectl get secret redis-auth -o yaml shows password: czNjcjN0LWV4YW1wbGU=, which is base64 for s3cr3t-example. Base64 is an encoding, not encryption: anyone allowed to read the Secret through the API can decode it with one command.

On disk it is plainer still. The API server stored the Secret in etcd as 557 bytes of protobuf, and the password appears there twice, in plain text. One copy is the raw bytes of the data field. The other sits in kubectl’s last-applied-configuration annotation: the manifest used stringData, and kubectl saves the whole manifest beside the object.

Encryption at rest is opt-in: an EncryptionConfiguration file on the API server with an aescbc, aesgcm or KMS v2 provider. It doesn’t help against someone who may create Pods in shop, because any Pod there can mount any Secret in the same namespace.

A Pod asks for storage with a PersistentVolumeClaim: a size and an access mode, but no backend. The StatefulSet created claim data-redis-0 for 1Gi ReadWriteOnce (one node at a time) from StorageClass standard. A StorageClass names a provisioner, here rancher.io/local-path, which creates PersistentVolumes, the objects that point at real storage.

standard sets volumeBindingMode: WaitForFirstConsumer, so no volume is made until a Pod using the claim is scheduled. The scheduler picked kind-worker 17 ms after the claim and wrote that choice onto it. The provisioner then created PersistentVolume pvc-247d41e1-…, a directory on kind-worker’s disk, 3,067 ms after the claim, and the binder bound the two together.

redis-0 got its Binding 4,022 ms after the claim, with its volume already waiting on the right node. Cloud disks and network storage plug in the same way, through CSI, the Container Storage Interface that storage drivers implement.

A StatefulSet names its Pods by ordinal instead of a random suffix: redis-0, then redis-1 if you scale up, created one at a time in order and deleted in reverse. For each ordinal it makes one claim from its volumeClaimTemplates, so redis-0 always mounts data-redis-0.

serviceName: redis points at a headless Service, one created with clusterIP: None. DNS for a headless Service returns the Pod IPs instead of one virtual IP, and each Pod gets a name of its own: redis-0.redis.shop.svc.cluster.local. Each version of the Pod template is kept as a ControllerRevision, here redis-75fc8b5997.

Deleting the StatefulSet leaves data-redis-0 and its volume in place by default. Apply the same manifest again and the new redis-0 mounts the old data.

Your 4 creates (Secret, ConfigMap, Service, StatefulSet) caused 22 more writes. The statefulset-controller wrote 6: a ControllerRevision, the claim, the Pod and three status updates. The persistent-volume-binder wrote 4 to bind claim and volume. The scheduler wrote 3: the node it nominated, the claim’s selected-node annotation and the Binding. The kubelet on kind-worker patched redis-0’s status 3 times.

The two endpoint controllers wrote 2 each: they created the address lists for Service redis, then added redis-0 once it was Ready. The pvc-protection-controller, which guards claims that a Pod still uses, wrote 1. Σ writes goes from 40 to 66.

The local-path-provisioner wrote the PersistentVolume. It is an ordinary Deployment in its own namespace that kind installs. Every other writer in this tutorial ships with Kubernetes.

set image patches one field, the container image in the Deployment’s Pod template (Σ 67). A different template hashes to a different pod-template-hash, so 9 ms later the deployment-controller creates a second ReplicaSet, web-d75cc9d86. A rolling update is that controller changing two numbers, a step at a time: the new ReplicaSet’s replicas go up while the old one’s go down.

The old ReplicaSet, web-67fddbbb7, is not deleted. It stays at 0 replicas, and the Deployment keeps up to revisionHistoryLimit (10) of them. kubectl rollout undo copies revision 1’s template back into the Deployment. That template hashes to 67fddbbb7 again, so the old ReplicaSet scales back up and no new one is made.

Two defaults bound the rollout: maxSurge 25%, how many Pods may exist above replicas, and maxUnavailable 25%, how many may be missing. For 3 replicas, 25% is 0.75. Surge rounds up, to 1, and unavailable rounds down, to 0. So this rollout may run 4 Pods at most and must keep 3 Ready throughout.

The deployment-controller worked in three rounds. It scaled web-d75cc9d86 up by one and waited for the new Pod to pass its readiness probe. Then it scaled web-67fddbbb7 down by one and the new set up by one, in the same moment. Rounds two and three each began when a v2 Pod turned Ready, at +4.1 s and +8.2 s.

The last v2 Pod was Ready at +9.2 s and the old ReplicaSet went to 0. rollout status returned at 20:12:19.082, 9.3 s after you ran the command. The rollout took 86 writes, and Σ writes reaches 152.

Each kubelet shows its node is alive by renewing a Lease, a small object in namespace kube-node-lease, every 10 s. kind-worker’s renewals landed at 20:11:43.085, 20:11:53.105, 20:12:03.482 and 20:12:13.677. At 20:12:19.835 the node stopped, and the next renewal never came.

The node-lifecycle controller, inside kube-controller-manager, waits out --node-monitor-grace-period, 50 s since v1.32 (40 s before). At 20:13:04.186, 50.5 s after the last renewal, it set the node’s Ready condition to Unknown, added the taint node.kubernetes.io/unreachable with effects NoSchedule and NoExecute, and marked the node’s Pods not Ready.

A Pod that is not Ready leaves its Service’s EndpointSlice: the endpointslice-controller removed pz7wt at 20:13:04.201. Until that write, kube-proxy on every node kept sending a third of new connections for web to an address where nothing answered.

A NoExecute taint evicts running Pods unless they tolerate it. Every Pod here carried two tolerations nobody wrote, for node.kubernetes.io/not-ready and unreachable, with tolerationSeconds: 300. The DefaultTolerationSeconds admission plugin adds them to every Pod at creation (chapter 01), so a Pod stays bound to an unreachable node for five minutes.

At 20:18:04.290, 300.1 s after the taint, the taint-eviction controller deleted the 3 Pods on kind-worker: v2 pz7wt, the old v1 pwq2w that was still terminating, and redis-0. Deleting a Pod for this reason is called eviction. On a node that can’t answer, it only marks the Pods for deletion.

12 ms later the replicaset-controller counted 2 Pods against replicas 3 and created web-d75cc9d86-dwv6l. The scheduler bound it to kind-worker2, the only worker left, which already had the image, and it was Ready at 20:18:05.698. Through all of it, web never had fewer than 2 Ready Pods.

The eviction marked redis-0 for deletion, but a Pod on an unreachable node stays Terminating until its kubelet confirms the containers are gone. A ReplicaSet doesn’t wait for that: dwv6l started while pz7wt was still Terminating. A StatefulSet does wait, because a second redis-0 could write to data-redis-0 while the first one might still be running.

It couldn’t have moved anyway: the PersistentVolume is a directory on kind-worker’s disk, pinned to that node by its node affinity. kind-worker restarted at 20:19:44.4 and was Ready at 20:19:46.2. Its kubelet confirmed the deletion at 20:19:48.018, and 6 ms later the statefulset-controller created a new redis-0, on kind-worker, Ready at 20:19:48.442.

Redis was down for 7 min 28.6 s, from the stop to that Ready. The node failure took 68 writes, and the last one, a StatefulSet status update, brought Σ writes to 220. Force-deleting the stuck Pod would have ended the wait early; the deep dive says why the docs warn against it.