Deep Dive into Kubernetes Agents: Kubelet and Kube-Proxy

In the complex architecture of a Kubernetes cluster, the Control Plane (Master) makes the decisions, but the Data Plane (Nodes) does the work. The critical components executing this work on every server are Kubelet and Kube-proxy.

While they often live in the shadow of high-profile components like the API Server or Scheduler, understanding these two agents is essential for debugging, networking, and maintaining cluster health. This guide breaks down their architecture, functionality, and configuration in 2025.

---

1. What is Kubelet? The "Node Agent"

The Kubelet is the core operational agent of Kubernetes. It runs as a system service (daemon) on every node in your cluster—whether that node is a physical server, a virtual machine, or a spot in the cloud. Its primary responsibility is to act on the instructions provided by the Kubernetes Control Plane.

Core Responsibilities

1. Pod Spec Execution: The Kubelet receives a Pod definition (via the API Server or local manifest file) and is responsible for ensuring that the containers described in that spec are running. It does not manage containers itself; it delegates this to a Container Runtime Interface (CRI) compliant runtime like containerd or CRI-O. 2. Health Monitoring: The Kubelet continuously monitors the health of the containers it manages. If a container crashes, the Kubelet restarts it based on the restartPolicy defined in the Pod spec (Always, OnFailure, Never). 3. Status Reporting: It acts as the heartbeat of the node. It gathers the status of the node (CPU, Memory, Disk pressure) and the Pods running on it, sending this data back to the API Server. This is how the Scheduler knows if a node has enough resources to accept new workloads.

How Kubelet Works (The Sync Loop)

The Kubelet operates using a control loop, often referred to as the Sync Loop.

1. Watch: It watches the API Server (or a local file directory) for Pod changes. 2. Sync: When a change is detected, it validates the spec and synchronizes the state. 3. Action: If a Pod needs to be created, the Kubelet: * Pulls the image (e.g., from Docker Hub or a private registry). * Instructs the CRI runtime to create the container. * Allocates resources (CPU/Memory requests).

The cAdvisor Integration

Historically, Kubelet integrated cAdvisor (Container Advisor) to auto-detect containers running on the node and gather resource usage statistics (CPU, memory, filesystem, network). In modern versions, cAdvisor is embedded directly into the Kubelet binary, exposing metrics on the default port 10250 (read-only) and 10255 (insecure).

---

2. What is Kube-Proxy? The Network Brain

While Kubelet manages the *compute*, Kube-proxy manages the *network connectivity* for Services. It runs on every node and ensures that network requests to a Kubernetes Service (a stable IP address) are correctly routed to a backend Pod, regardless of where that Pod is located in the cluster.

The Problem Kube-Proxy Solves

Pods in Kubernetes are ephemeral. Their IP addresses change constantly when they restart or are rescheduled. Services provide a stable abstraction (DNS name/IP) for these dynamic Pods. Kube-proxy manages the underlying network rules to make this abstraction work.

Kube-Proxy Modes in 2025

Kube-proxy implements its functionality through different modes. The mode dictates how network traffic is actually redirected to the backend Pods.

A. iptables Mode (Legacy)

In this mode, kube-proxy sets up iptables rules on the Linux kernel. When a packet hits the Service IP, the kernel matches it against a rule and probabilistically sends it to a backend Pod.

• Pros: Stable, kernel-native.
• Cons: Random load balancing (not smooth), performance issues with thousands of services because iptables is linear (checking every rule until a match is found).

B. IPVS Mode (IP Virtual Server) - High Performance

Recommended for production environments with heavy traffic. IPVS uses Netfilter hooks to direct traffic. It supports multiple load-balancing algorithms (Round Robin, Least Connection, etc.).

• Pros: O(1) lookup time (constant speed regardless of service count), better load balancing.
• Cons: Requires ip_vs kernel modules loaded on the host.

C. nftables (The Future)

With the deprecation of iptables in modern Linux distros in favor of nftables, newer versions of Kube-proxy are moving toward nftables support, offering a more unified packet filtering framework.

D. Userspace Mode (Deprecated)

Historically, Kube-proxy would open a port in userspace and proxy the traffic there. It is slow and rarely used today.

---

3. Kubelet vs. Kube-Proxy: A Technical Comparison

It is crucial to distinguish these two agents. While they run side-by-side, they occupy different layers of the stack.

| Feature | Kubelet | Kube-proxy | | :--- | :--- | :--- | | Primary Function | Compute & Lifecycle Management | Network Routing & Load Balancing | | API Focus | Pods, Nodes, Volumes | Services, Endpoints | | Protocol | Uses HTTP (REST) to talk to API Server | Watch API for Services/Endpoints updates | | Data Path | Interfaces with Container Runtime (CRI) | Interfaces with Netfilter (iptables/IPVS) | | Failure Impact | Pods stop running; Node marked NotReady | Network connectivity to Services fails | | Key Metrics | CPU/Memory usage, Pod Restart Count | Connections, Bytes Transferred |

---

4. Technical Implementation & Python Examples

Inspecting Kubelet Metrics

The Kubelet exposes metrics (often used by Prometheus) on port 10250 (secure) or 10255 (read-only). You can query the /metrics/cadvisor endpoint to inspect container resource usage.

Here is a Python example using requests to fetch raw metrics from a local Kubelet endpoint (assuming authentication is bypassed for this example):

import requests
In a real cluster, this requires SSL certs and Token auth
configured in ~/.kube/config or passed as headers.
KUBELET_URL = "http://localhost:10255/metrics/cadvisor"
def get_pod_memory_usage(): try: response = requests.get(KUBELET_URL) response.raise_for_status()
# Filter for container memory usage for line in response.text.split('\n'): if 'container_memory_usage_bytes' in line and 'pod' in line: print(line)
except Exception as e: print(f"Error connecting to Kubelet: {e}")
if __name__ == "__main__": get_pod_memory_usage() 

Verifying Kube-Proxy Rules

Kube-proxy leaves footprints on the host operating system. You can verify the rules it writes by inspecting iptables or ipvsadm.

If you SSH into a Kubernetes node, you can list the IPVS services managed by Kube-proxy:

Shows the virtual service IPs (ClusterIPs) mapped to real server IPs (Pod IPs)
sudo ipvsadm -Ln 

Or view the NAT table rules for a specific Service (e.g., Kubernetes default service):

List rules in the NAT table for KUBE-SERVICES
sudo iptables -t nat -L KUBE-SERVICES | grep '10.96.0.1' 

---

5. Modern Context: Edge Computing and K3s

In 2025, Kubernetes is not just for massive data centers. It has moved to the edge (IoT, Edge computing).

When looking at K3s (a lightweight Kubernetes distribution), the role of Kube-proxy is often replaced or augmented by: 1. Klipper LB: K3s uses a lightweight load balancer implementation called Klipper, which leverages nftables (or iptables) to provide similar functionality to Kube-proxy but with a much smaller memory footprint. 2. CNI Plugins: In many "k3s kube-proxy" discussions, users explore disabling the default Kube-proxy in favor of CNI-specific plugins (like Cilium), which use eBPF to handle load balancing entirely in the kernel without iptables or kube-proxy, offering significantly higher performance.

---

6. Troubleshooting Common Issues

"Kubelet is unhealthy"

If your nodes are marked NotReady, Kubelet is usually the culprit.

• Check Logs: journalctl -u kubelet -f
• Common Cause: Swap Memory. Kubelet fails to start if swap is enabled (a design choice by Kubernetes to ensure stable QoS). You must disable swap with swapoff -a.

"Services not responding (Connection Refused)"

If Pods are running, but you cannot reach the Service IP:

• Check Kube-proxy: Ensure the kube-proxy pods are running in the kube-system namespace.
• Check Mode: Are you mixing IPVS and iptables?
• Check CNI: Does the CNI plugin conflict with the ports Kube-proxy is trying to bind to?

---

Conclusion

Kubelet and Kube-proxy are the "hands and feet" of Kubernetes.

• Kubelet is the hands: It manipulates the container runtime, pulling images and starting/stopping processes based on the desires of the Control Plane.
• Kube-proxy is the feet: It moves the network traffic, creating complex rules on the OS kernel to ensure that traffic reliably flows into the dynamic microservices managed by Kubelet.

Understanding how these two interact is the difference between a junior operator who restarts things blindly and a senior engineer who can architect resilient, high-performance clusters.