what actually happens after kubectl apply

Kubernetes looks like one big machine from the outside. You hand it a YAML file, and a few seconds later your app is running somewhere in a cluster. It's tempting to picture a single "deploy" function that reads your file and starts containers.

That function doesn't exist. Kubernetes is a group of small, independent programs. Each one watches a shared database for something it cares about, does one job, and writes the result back. None of them calls the others directly. Once that clicks, most of Kubernetes stops feeling like magic.

This post follows a single Deployment from kubectl apply to a running container, one component at a time.

Foundation

The most important idea in Kubernetes is that it is declarative. You never say "start three containers". You say "there should be three of these", and the cluster keeps working until that's true. If something drifts later (a node dies, someone deletes a pod), it keeps working until it's true again.

So everything you apply is a record of desired state. Each object also has a status that holds the observed state. Nearly every component in the cluster is a loop that compares the two and nudges reality toward what you asked for.

Here's the file we'll follow:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: web
spec:
  replicas: 3
  selector:
    matchLabels:
      app: web
  template:
    metadata:
      labels:
        app: web
    spec:
      containers:
        - name: web
          image: nginx:1.27
          ports:
            - containerPort: 80
          readinessProbe:
            httpGet:
              path: /
              port: 80

The request

kubectl apply -f web.yaml

kubectl reads your kubeconfig to find the cluster and your credentials. It converts the YAML to JSON and sends it to the API server as a plain HTTPS request. kubectl itself has no special access to the cluster. Anything it does, you could do with curl and the right certificate.

Before the API server saves anything, the request goes through three gates:

1. Authentication: who are you? Usually a client certificate or a token.
2. Authorization: are you allowed to create Deployments in this namespace? This is where RBAC rules apply.
3. Admission: admission controllers can change the object (mutating) or reject it (validating). This is how default resource limits get filled in, or how a policy engine blocks images from registries you don't trust.

If the request passes all three, the object is written to etcd, a consistent key-value store that holds the entire state of the cluster. You get back 201 Created.

Now for the part that surprised me most the first time: at this point nothing is running. No pod, no container, not even a decision about where one should go. All Kubernetes has done is write down that you want a Deployment called web. Everything from here on happens because other components notice that record.

Controllers

Nobody tells the next component to act. The components learn about changes by watching the API server. A watch is a long-lived request that streams every change to a type of object as it happens. Each component watches the types it cares about and reacts.

The first to react is the Deployment controller, which runs inside kube-controller-manager. It sees a new Deployment with no ReplicaSet, so it creates one. A ReplicaSet is the object that actually says "keep N copies of this exact pod template running". The Deployment sits on top of it to handle rollouts: when you change the image, it creates a new ReplicaSet and gradually moves replicas from the old one to the new one.

The ReplicaSet controller sees the new ReplicaSet. It wants 3 pods and counts 0, so it creates three Pod objects.

Both controllers follow the same pattern, called the reconcile loop:

This is also why Kubernetes heals itself. Delete one of those pods by hand and the ReplicaSet controller sees 2 where it wants 3, so it creates another. Nothing special happens for failures. The same loop that created the pods in the first place notices the gap and fills it.

A controller doesn't need to remember what it did last time either. It only looks at the current state and asks how far it is from the desired state. If a controller crashes halfway through, the one that replaces it reads the same state and picks up where it left off.

Scheduling

The three Pod objects now exist, but each has an empty spec.nodeName. A pod that isn't assigned to a node can't run anywhere.

The scheduler watches for exactly these pods. For each one it does two things:

• Filter: drop every node where the pod can't run. Not enough free CPU or memory for the pod's requests, a taint the pod doesn't tolerate, a nodeSelector that doesn't match.
• Score: rank the nodes that are left. Prefer the one that spreads replicas across nodes or zones, already has the image cached, or balances the load.

The scheduler then writes a binding back to the API server, which sets nodeName: node-2. That's the whole decision. The scheduler doesn't start anything either. It writes one field and moves on to the next pod.

Running

Each node runs an agent called the kubelet. It watches for pods whose nodeName is its own node. When web-7d4b9-p9w2m gets bound to node-2, node-2's kubelet picks it up and does the actual work:

1. It asks the container runtime (usually containerd) over the Container Runtime Interface (CRI) to pull nginx:1.27, unless the node already has the image.
2. It has the CNI plugin give the pod a network namespace and an IP address.
3. It starts the container and begins running the readiness probe.
4. It reports the pod's status (Running, then Ready) back to the API server.

Pulling the image is usually the slowest step. A small image the node already has can start in under a second. A multi-gigabyte image on a brand-new node can take minutes. If your pods sit in ContainerCreating for a long time, this is usually why.

Once the readiness probe passes, the pod is marked Ready. Another controller (the EndpointSlice controller) sees that and adds the pod's IP to any Service whose selector matches app: web. Only then does the pod receive traffic. If you've ever wondered why a pod showed Running but got no requests, check whether it's Ready.

Putting it together

Counting all the steps, one kubectl apply involved:

None of these components knows the others exist. They only share the API server. That's why you can replace the scheduler, add your own controller for a custom resource, or lose a node without the rest of the system noticing. It's also the idea behind operators: an operator is a controller you write yourself, using the same loop as every built-in one.

Further reading

• Cluster Architecture, the official overview of every component in this post
• Controllers, the reconcile loop in the official docs' own words
• Kubernetes Scheduler, filtering and scoring in more depth
• Admission Control, everything that can change or reject your object before it's stored
• what-happens-when-k8s, this same journey traced through the actual source code
• Borg, Omega, and Kubernetes, the paper on where these ideas came from at Google