ADR-001: Cilium as CNI over Calico

Status: Accepted Date: 2025-06-15 Author: Spencer Fuller

Context

I needed a Container Network Interface (CNI) plugin for a 4-node ARM64 Kubernetes cluster built on Orange Pi 5 single-board computers (Rockchip RK3588, 8 CPU / 16GB RAM each). The cluster runs mixed workloads: Home Assistant for IoT/home automation, AI agent infrastructure (OpenClaw), security scanning CronJobs, and distributed storage (Longhorn). Kubernetes v1.28.2 via kubeadm.

Key requirements:

• Resource efficiency — SBCs have limited CPU and memory; the CNI can’t be a resource hog
• L7-aware network policies — IoT workloads (Home Assistant) need granular segmentation from agent workloads and the broader network. L3/L4 policies alone aren’t sufficient when HA uses HTTP/WebSocket APIs that need per-path control
• Network observability — visibility into pod-to-pod and pod-to-external traffic for security monitoring and debugging
• ARM64 support — first-class, not afterthought

Decision

Adopt Cilium with eBPF as the cluster CNI, replacing the default kube-proxy with Cilium’s eBPF-based kube-proxy replacement.

Rationale

1. eBPF-based dataplane eliminates iptables overhead. On resource-constrained SBCs, iptables rule processing scales poorly as services grow. Cilium’s eBPF programs attach directly to the kernel networking stack, providing O(1) lookups instead of O(n) iptables chain traversal. This matters when you have 8 cores and every cycle counts.

2. CiliumNetworkPolicy enables L7-aware segmentation. Standard Kubernetes NetworkPolicy operates at L3/L4 only. CiliumNetworkPolicy supports HTTP-aware rules — I can allow Home Assistant to receive HTTP traffic on specific paths while blocking everything else, or restrict which agents can call which API endpoints. This is critical for IoT segmentation where the threat model includes compromised smart devices making unexpected API calls.

3. Hubble provides network observability out of the box. Hubble (Cilium’s observability layer) gives real-time visibility into network flows, DNS queries, and HTTP requests across the cluster. This feeds directly into security monitoring — I can see exactly what Home Assistant is talking to, which pods are making external requests, and whether any unexpected traffic patterns emerge. No separate tool needed.

4. Strong ARM64 support. Cilium publishes multi-arch container images and the eBPF programs compile at runtime for the target architecture. The vendor kernel (6.1.115-vendor-rk35xx) includes eBPF support, which was verified before deployment.

Alternatives Considered

Alternative	Why Not
Calico	Solid, mature CNI with good community. However, Calico’s dataplane is iptables-based (eBPF dataplane was experimental at evaluation time). NetworkPolicy support is standard K8s L3/L4 only without Calico Enterprise. Less granular than CiliumNetworkPolicy for IoT segmentation.
Flannel	Simple VXLAN overlay — great for getting started, but no NetworkPolicy support at all. No observability. Too basic for a cluster with IoT and security requirements.
Weave Net	Declining community activity and maintenance. Encryption built-in is nice, but not worth the risk of adopting a project with uncertain long-term support.

Consequences

Positive

• eBPF kube-proxy replacement reduced CPU overhead measurably on the Orange Pi 5 nodes
• L7-aware CiliumNetworkPolicies enabled fine-grained IoT segmentation (e.g., Home Assistant’s hostNetwork requirement was contained with toEntities: ["remote-node"] policies)
• Hubble provides network flow data usable by the cybersecurity scanning pipeline

Negative

• Learning curve for CiliumNetworkPolicy CRDs — the syntax is more complex than standard NetworkPolicy, and debugging policy drops requires familiarity with cilium monitor and Hubble
• Kernel dependency — Cilium requires eBPF support in the kernel. The vendor Rockchip kernel (6.1.115) has it, but kernel upgrades need to be validated against Cilium compatibility
• Hubble resource overhead — the Hubble relay and UI consume additional memory across nodes; this was acceptable but required tuning resource limits on the smaller nodes