Architecture, the API and Reconciliation
Kubernetes is a declarative system: you store desired state through the API server, etcd keeps it, and controllers reconcile actual state towards it. Know which component does what, how the API handles concurrency and deletion, and what survives a control-plane outage.
Key points
- 1
kube-apiserver is the only etcd client; every component and kubectl goes through it, and every request passes authentication, authorization and admission.
- 2
etcd needs a majority: floor(n/2) + 1. Run 3 or 5 members; 2 or 4 add machines without adding failure tolerance.
- 3
Controllers are level-triggered reconcile loops: idempotent, robust to missed events, and they revert drift from the desired spec.
- 4
Optimistic concurrency: a write with a stale resourceVersion fails with 409 Conflict; a watch from a compacted version gets 410 Gone and must relist.
- 5
Deletion is two-phase when finalizers exist: deletionTimestamp is set and the object stays until every finalizer is removed. ownerReferences drive cascading garbage collection.
- 6
Version skew: kubelets may be up to three minor versions older than kube-apiserver but never newer; upgrade the control plane first, then nodes.
- 7
When the control plane is down, the data plane keeps running: containers serve, kubelets restart crashes, Service rules route. Nothing that needs an API write happens.
Common traps
Adding a second control-plane node with stacked etcd lowers fault tolerance: 2 members tolerate zero failures.
An inequality label selector (tier!=frontend) also matches objects that don't have the label at all.
Removing finalizers by hand makes a stuck object disappear but skips the cleanup the finalizer was protecting.
Read the source
Test yourself on Architecture, the API and Reconciliation
Ten questions, with the answer and explanation after each one.