Kubernetes the Hard Way

If you've worked in modern cloud infrastructure, you've likely encountered Kubernetes. It powers everything from startup deployments to global-scale platforms, yet many engineers interact with it only through YAML manifests and deployment pipelines.

While managed Kubernetes services simplify operations, they often hide the underlying mechanics that make Kubernetes work.

This is where Kubernetes the Hard Way becomes invaluable.

Originally created by Kelsey Hightower, Kubernetes the Hard Way is not simply a tutorial. It is a deep dive into the internal architecture of Kubernetes, teaching engineers how each component interacts by requiring them to build a cluster manually from the ground up.

By the end, you'll understand not only how to use Kubernetes, but why Kubernetes works.

Why Learn Kubernetes the Hard Way?

Modern cloud providers make it possible to create a Kubernetes cluster with a single command.

For example:

eksctl create cluster

gcloud container clusters create production

While convenient, these abstractions hide critical concepts:

Certificate management
Control plane communication
Service discovery
Cluster networking
Scheduler behavior
Worker node registration
etcd operations

Understanding these components transforms Kubernetes from a black box into an understandable distributed system.

Engineering Insight: Engineers who understand Kubernetes internals troubleshoot production issues significantly faster than those who only interact with high-level abstractions.

What Is Kubernetes?

Kubernetes is a container orchestration platform designed to automate:

Deployment
Scaling
Networking
Service discovery
Self-healing
Resource management

At a high level:

Users
  │
  ▼
Kubernetes API
  │
  ▼
Control Plane
  │
  ▼
Worker Nodes
  │
  ▼
Containers

The control plane makes decisions.

The worker nodes execute workloads.

Cluster Architecture Overview

Before building a cluster, it is important to understand its major components.

Control Plane Components

The control plane acts as the brain of Kubernetes.

It consists of:

API Server
Scheduler
Controller Manager
etcd

            API Server
                 │
        ┌────────┼────────┐
        ▼        ▼        ▼
 Scheduler  Controllers  etcd

Each component serves a specific purpose.

Worker Node Components

Worker nodes run application workloads.

A worker node contains:

kubelet
kube-proxy
Container Runtime

Worker Node
│
├── kubelet
├── kube-proxy
└── Container Runtime

These services communicate with the control plane continuously.

Step 1: Provisioning Infrastructure

The first phase of Kubernetes the Hard Way is creating infrastructure manually.

Typical setup:

Resource	Count
Control Plane Nodes	3
Worker Nodes	3
Load Balancer	1
Network	1

Unlike managed Kubernetes, nothing is preconfigured.

You are responsible for every component.

Step 2: Certificate Authority and TLS

One of the most eye-opening parts of the project is certificate generation.

Every Kubernetes component communicates securely using TLS.

You'll generate certificates for:

API Server
kubelet
Controller Manager
Scheduler
Service Accounts
Administrators

Example:

openssl genrsa -out ca.key 4096

Create the certificate:

openssl req -x509 \
  -new \
  -sha512 \
  -days 3650 \
  -key ca.key \
  -out ca.crt

At this stage, Kubernetes starts looking less like a deployment platform and more like a distributed security system.

Step 3: Building etcd

Kubernetes stores all cluster state in etcd.

Examples include:

Pods
Deployments
Services
Secrets
ConfigMaps

Without etcd, Kubernetes has no memory.

Architecture:

            etcd Cluster
        ┌──────┼──────┐
        ▼      ▼      ▼
      Node1  Node2  Node3

High availability requires multiple etcd members.

Example startup command:

etcd \
  --name controller-0 \
  --data-dir=/var/lib/etcd

Step 4: Installing the API Server

The Kubernetes API Server is the central communication hub.

Everything interacts with it:

kubectl
kubelet
Scheduler
Controllers

Request flow:

kubectl
    │
    ▼
API Server
    │
    ▼
etcd

The API Server validates requests and stores cluster state.

Example:

kube-apiserver \
  --advertise-address=10.0.0.1 \
  --etcd-servers=https://127.0.0.1:2379

Step 5: Configuring the Scheduler

The scheduler decides where workloads should run.

Example scenario:

Pod Request
      │
      ▼
Scheduler
      │
      ▼
Best Available Node

The scheduler evaluates:

CPU requirements
Memory requirements
Affinity rules
Taints and tolerations
Resource availability

Without the scheduler, Pods remain in a Pending state indefinitely.

Step 6: Controller Manager

Controllers constantly monitor desired versus actual state.

For example:

Desired State:

3 Pods Running

Actual State:

2 Pods Running

Controller Action:

Create 1 New Pod

This reconciliation loop is one of Kubernetes' most powerful design patterns.

Example startup:

kube-controller-manager \
  --cluster-name=kubernetes

Step 7: Bootstrapping Worker Nodes

Worker nodes are where applications actually run.

Each node runs kubelet.

Responsibilities include:

Pulling container images
Starting containers
Reporting node health
Managing Pods

Example:

kubelet \
  --config=kubelet-config.yaml

The kubelet continuously communicates with the API Server.

Step 8: Container Runtime

Containers require a runtime.

Historically:

Docker
containerd
CRI-O

Modern Kubernetes primarily uses containerd.

Architecture:

Pod
 │
 ▼
containerd
 │
 ▼
Linux Kernel

The runtime performs actual container execution.

Step 9: Cluster Networking

Networking is often the most challenging part of Kubernetes.

Requirements:

Pod-to-Pod communication
Pod-to-Service communication
External access

Example flow:

Pod A
  │
  ▼
Pod B

Pods should communicate without NAT.

Common networking solutions:

Calico
Flannel
Cilium
Weave

Architectural Note: Kubernetes networking is intentionally flexible, allowing organizations to choose networking implementations that fit their requirements.

Step 10: DNS and Service Discovery

Applications need stable endpoints.

Pods are ephemeral:

pod-123
pod-456
pod-789

Instead of using Pod IPs directly:

api-service.default.svc.cluster.local

CoreDNS provides service discovery.

Example:

apiVersion: v1
kind: Service
metadata:
  name: api-service

Applications can discover each other without knowing underlying IP addresses.

Step 11: Deploying a Test Application

Once the cluster is operational, deploy a workload.

Example Deployment:

apiVersion: apps/v1
kind: Deployment

metadata:
  name: nginx

spec:
  replicas: 3

  selector:
    matchLabels:
      app: nginx

  template:
    metadata:
      labels:
        app: nginx

    spec:
      containers:
      - name: nginx
        image: nginx:latest

Apply:

kubectl apply -f nginx.yaml

Verify:

kubectl get pods

If everything works, you've successfully built Kubernetes from scratch.

Key Lessons Learned

Completing Kubernetes the Hard Way reveals several important truths.

Kubernetes Is Fundamentally an API System

Everything revolves around API objects.

Pods, Deployments, Services, and ConfigMaps are simply resources stored in etcd.

Security Is Built Into Every Layer

TLS certificates are everywhere.

Every component authenticates and authorizes requests.

Kubernetes Is a Distributed System

Many operational challenges stem from distributed systems principles:

Consensus
Fault tolerance
Eventual consistency
Network failures

Controllers Drive Everything

Kubernetes operates through continuous reconciliation loops.

Desired state is more important than individual commands.

Common Mistakes During the Lab

Many engineers encounter these issues:

Incorrect Certificates

Symptoms:

x509 certificate signed by unknown authority

Usually caused by:

Wrong Subject Alternative Names
Incorrect certificate authority configuration

etcd Connectivity Issues

Symptoms:

Failed to connect to etcd

Typically caused by:

Firewall rules
TLS misconfiguration
Incorrect peer URLs

Node Registration Failures

Symptoms:

kubectl get nodes

No resources found

Usually indicates kubelet authentication problems.

Is Kubernetes the Hard Way Still Worth It?

Absolutely.

Even though most production environments use managed services such as:

Amazon EKS
Google GKE
Azure AKS

The knowledge gained remains extremely valuable.

You'll better understand:

Why clusters fail
How networking works
How control plane components communicate
How security is implemented
How workloads are scheduled

Most importantly, you'll develop a mental model that helps diagnose complex production issues.

Conclusion

Kubernetes the Hard Way is one of the most educational exercises a DevOps or Platform Engineer can undertake.

It strips away managed-service abstractions and exposes the building blocks of Kubernetes itself. By manually provisioning infrastructure, generating certificates, configuring etcd, deploying control plane components, and bootstrapping worker nodes, engineers gain a deep understanding of how the platform operates internally.

While few teams will ever build production clusters this way, the lessons learned translate directly into real-world operations, troubleshooting, and architecture design.

The best Kubernetes administrators are not simply experts at writing YAML—they understand the distributed system running beneath it. Kubernetes the Hard Way provides exactly that understanding.