Kubernetes Cluster Bootstrap and Cilium Validation

PURPOSE

The build established a real Kubernetes foundation for the lab.

This build established the first functional Kubernetes foundation in the lab. The immediate objective was to bring up a small but realistic cluster that could later support ingress, service mesh testing, observability, TLS experiments, application deployment, and operational troubleshooting.

The goal was not to hide behind a one-click distribution. The point was to build enough of the stack manually to understand where Kubernetes depends on the operating system, container runtime, node networking, CNI behavior, and cluster bootstrap process.

ENVIRONMENT

The cluster started as three prepared Ubuntu virtual machines.

Cluster Layout

Three Ubuntu 24.04 virtual machines
One control plane node
Two worker nodes
Static host configuration across nodes
Cluster built inside the lab network boundary

Core Components

kubeadm for cluster bootstrap
kubelet and kubectl for node and cluster control
containerd as the container runtime
Helm for Cilium deployment
Cilium as the initial CNI

ARCHITECTURE FLOW

The cluster was built layer by layer.

flowchart TB
  LAB[Proxmox Lab]

  CP[Control Plane]
  W1[Worker 1]
  W2[Worker 2]

  KUBE[Kubernetes Tools]
  RUNTIME[containerd]
  HELM[Helm]
  CNI[Cilium]
  PODS[Pods]
  TESTS[Validation]

  LAB --> CP
  LAB --> W1
  LAB --> W2

  CP --> KUBE
  W1 --> KUBE
  W2 --> KUBE

  KUBE --> RUNTIME
  KUBE --> HELM
  HELM --> CNI
  CNI --> PODS
  PODS --> TESTS

KEY DECISIONS

The design choices prioritized learning, repeatability, and future platform work.

kubeadm Instead of a Shortcut

kubeadm was used because the objective was to understand the bootstrap path instead of abstracting it away. This exposed the relationship between node preparation, container runtime configuration, kubelet behavior, and CNI readiness.

containerd Runtime

containerd was selected as the runtime because it aligns with the modern Kubernetes operating model. The runtime was configured and validated before relying on kubeadm so runtime problems would not be confused with cluster-level failures.

Systemd Cgroups

The container runtime was aligned with systemd cgroup behavior to match kubelet expectations. This was treated as a foundational node configuration item instead of something to troubleshoot later after the cluster was already unstable.

Cilium as the Initial CNI

Cilium was chosen over simpler CNI options because it better supports the direction of the lab and mirrors the type of networking work being evaluated professionally. It also provides a stronger path toward future eBPF-based networking, observability, and policy work.

CNI TRADEOFF

Cilium was selected for the direction the lab is going, not just the fastest path.

The CNI decision was one of the most important choices in the build. Flannel would have been simple and fast, but it would not provide the same future path for policy, observability, or deeper networking behavior. Calico remains a strong and proven option, especially for network policy use cases, but Cilium matched the lab direction better because of its eBPF foundation and its relevance to the environments being evaluated at work.

The practical decision was to accept a slightly more advanced CNI early because the lab is meant to become a real testing and learning platform, not just a minimal cluster that passes a basic smoke test.

IMPLEMENTATION PATH

The work moved from node preparation to cluster validation.

Prepared three Ubuntu virtual machines for cluster roles.
Configured node identity, host resolution, and static addressing.
Installed and configured containerd.
Aligned containerd cgroup behavior with kubelet expectations.
Installed Kubernetes tooling across nodes.
Initialized the control plane with kubeadm.
Joined both worker nodes to the cluster.
Installed Cilium through Helm.
Validated cluster health and Cilium readiness.
Ran Cilium validation testing and confirmed successful results.

TROUBLESHOOTING NOTES

The troubleshooting process stayed focused on one layer at a time.

The troubleshooting process stayed focused on validating one layer at a time. Node configuration came first, then container runtime behavior, then kubeadm bootstrap, then CNI installation. That prevented the build from turning into a blind pile of changes.

Node Readiness

Hostname and address consistency mattered. Kubernetes becomes noisy quickly when node identity, static IPs, or local host resolution are wrong. Those basics were validated before moving deeper into the stack.

Runtime Before Cluster

containerd needed to be running and configured correctly before the cluster bootstrap could be trusted. The runtime was treated as a dependency, not an afterthought.

CNI as a Gate

The cluster was not considered healthy just because nodes joined. Pod networking had to be functional before moving forward. Cilium status and test validation were used as the gate.

Avoiding Stack Creep

Storage, ingress, service mesh, and observability were intentionally deferred until the base cluster was validated. The first milestone was a healthy networked cluster, not a pile of half-working add-ons.

VALIDATION

The cluster was validated with node checks, Cilium health, and Cilium testing.

The cluster was validated by confirming node membership, checking Cilium health, and running the Cilium validation test suite. The final validation result was a successful Cilium test run with 132 checks passing.

Control Plane

Initialized successfully and became the management point for the cluster.

Worker Nodes

Both worker nodes joined successfully and appeared in cluster node output.

Cilium

Cilium installed through Helm and reported healthy status after deployment.

Test Result

Cilium validation completed successfully with 132 checks passing.

OUTCOME

The result was a validated three-node Kubernetes foundation.

The result was a functional three-node Kubernetes cluster built from zero in roughly 5.5 hours. That included VM preparation, Kubernetes bootstrap, container runtime configuration, CNI selection, Cilium installation, and validation testing.

More importantly, the build created a repeatable foundation for future lab work. The cluster is now positioned to support ingress testing, Istio or other service mesh work, observability, TLS experiments, workload deployment, and operational troubleshooting.

LESSONS LEARNED

The base platform needs to be healthy before adding more layers.

A Kubernetes cluster is only as stable as the layers underneath it: node configuration, runtime configuration, and networking all matter.
CNI selection should match the future direction of the environment, not just the fastest path to a green cluster.
Successful node joins are not the finish line. Pod networking and CNI validation are what make the cluster usable.
Deferring storage, ingress, and service mesh until the base cluster is healthy keeps troubleshooting controlled.
Time tracking matters. Being able to build a validated networked cluster from zero in roughly 5.5 hours is a useful engineering benchmark.

NEXT ACTIONS

The validated base cluster becomes the platform for the next phase.

Add an ingress or gateway layer for application access testing.
Continue service mesh evaluation with Istio as the next direction.
Add observability tooling once workload routing is stable.
Document TLS and certificate handling patterns.
Evaluate storage options after the base platform remains stable.
Continue writing operator logs as major lab capabilities are added.