Skip to content

Cilium & eBPF Networking - Primer

Why This Matters

Who made it: Cilium was created by Thomas Graf and Daniel Borkmann at Isovalent in 2016. Both are Linux kernel network subsystem maintainers — Borkmann is one of the primary eBPF maintainers in the kernel. Isovalent was acquired by Cisco in 2023. Cilium became a CNCF Incubating project in 2021 and graduated in October 2023, making it the first eBPF-based project to reach CNCF Graduation.

Name origin: Cilium is named after the biological cilia — tiny hair-like structures on cell surfaces that sense and interact with their environment. The metaphor: Cilium attaches eBPF programs to the kernel's networking "surface" to observe and control network traffic at the most fundamental level.

Cilium is the eBPF-powered CNI (Container Network Interface) that is replacing iptables in Kubernetes. Traditional CNIs like Calico and Flannel use iptables or IPVS for packet routing and policy enforcement. At scale, iptables rule chains become a performance bottleneck — every packet walks a linear chain of rules. Cilium uses eBPF programs attached directly to the Linux kernel's networking stack, bypassing iptables entirely. This gives you faster packet processing, identity-based network policies (not just IP-based), built-in observability via Hubble, transparent encryption, and service mesh capabilities without sidecar proxies. Cilium is the default CNI in GKE, EKS Anywhere, and is a CNCF graduated project.

Core Concepts

1. Installation

# Install Cilium CLI
CILIUM_CLI_VERSION=$(curl -s https://raw.githubusercontent.com/cilium/cilium-cli/main/stable.txt)
curl -L --fail --remote-name-all \
  https://github.com/cilium/cilium-cli/releases/download/${CILIUM_CLI_VERSION}/cilium-linux-amd64.tar.gz
sudo tar xzvf cilium-linux-amd64.tar.gz -C /usr/local/bin

# Install Cilium into a cluster
cilium install --version 1.15.0

# With specific options
cilium install --version 1.15.0 \
  --set kubeProxyReplacement=true \
  --set hubble.enabled=true \
  --set hubble.relay.enabled=true \
  --set hubble.ui.enabled=true

# Or via Helm
helm repo add cilium https://helm.cilium.io/
helm install cilium cilium/cilium --version 1.15.0 \
  --namespace kube-system \
  --set hubble.relay.enabled=true \
  --set hubble.ui.enabled=true

# Verify installation
cilium status --wait
cilium connectivity test    # runs a full connectivity test suite

2. Network Policies

Cilium extends Kubernetes NetworkPolicy with CiliumNetworkPolicy, which supports L7 rules, DNS-based policies, and identity-aware filtering.

Standard Kubernetes NetworkPolicy: 

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-frontend-to-backend
  namespace: production
spec:
  podSelector:
    matchLabels:
      app: backend
  policyTypes:
    - Ingress
  ingress:
    - from:
        - podSelector:
            matchLabels:
              app: frontend
      ports:
        - protocol: TCP
          port: 8080

CiliumNetworkPolicy with L7 rules: 

apiVersion: cilium.io/v2
kind: CiliumNetworkPolicy
metadata:
  name: api-access-control
  namespace: production
spec:
  endpointSelector:
    matchLabels:
      app: api-server
  ingress:
    - fromEndpoints:
        - matchLabels:
            app: frontend
      toPorts:
        - ports:
            - port: "8080"
              protocol: TCP
          rules:
            http:
              - method: "GET"
                path: "/api/v1/public/.*"
              - method: "POST"
                path: "/api/v1/orders"
                headers:
                  - 'X-Request-ID: .*'

DNS-based policy (allow egress only to specific domains): 

apiVersion: cilium.io/v2
kind: CiliumNetworkPolicy
metadata:
  name: allow-external-apis
  namespace: production
spec:
  endpointSelector:
    matchLabels:
      app: payment-service
  egress:
    - toFQDNs:
        - matchName: "api.stripe.com"
        - matchName: "api.paypal.com"
      toPorts:
        - ports:
            - port: "443"
              protocol: TCP
    - toEndpoints:
        - matchLabels:
            "k8s:io.kubernetes.pod.namespace": kube-system
            "k8s:k8s-app": kube-dns
      toPorts:
        - ports:
            - port: "53"
              protocol: UDP
          rules:
            dns:
              - matchPattern: "*.stripe.com"
              - matchPattern: "*.paypal.com"

3. Hubble Observability

Hubble provides network observability built into Cilium — flow logs, service maps, and DNS visibility without installing additional agents.

# Install Hubble CLI
HUBBLE_VERSION=$(curl -s https://raw.githubusercontent.com/cilium/hubble/main/stable.txt)
curl -L --fail --remote-name-all \
  https://github.com/cilium/hubble/releases/download/$HUBBLE_VERSION/hubble-linux-amd64.tar.gz
sudo tar xzvf hubble-linux-amd64.tar.gz -C /usr/local/bin

# Port-forward Hubble Relay
cilium hubble port-forward &

# Check Hubble status
hubble status

# Observe all flows
hubble observe

# Filter by namespace
hubble observe --namespace production

# Filter by pod
hubble observe --pod production/api-server-abc123

# Filter by verdict (dropped packets — policy violations)
hubble observe --verdict DROPPED

# Filter by HTTP status
hubble observe --http-status 500

# Filter by DNS queries
hubble observe --protocol dns

# Filter by traffic direction
hubble observe --type trace:to-endpoint    # ingress
hubble observe --type trace:from-endpoint  # egress
hubble observe --type drop                 # drops only

# JSON output for processing
hubble observe --namespace production --output json | jq '.flow.source.labels'

# Flow count summary
hubble observe --namespace production --last 5m -o compact | wc -l

# Access Hubble UI (service map visualization)
cilium hubble ui
# Opens browser to http://localhost:12000

Under the hood: Cilium assigns a numeric identity to each unique set of pod labels. Network policy enforcement happens against these identities, not IP addresses. This is a fundamental difference from iptables-based CNIs: when a pod restarts with a new IP, iptables rules must be regenerated, but Cilium's identity-based rules remain valid. This is why Cilium handles large-scale pod churn better than traditional CNIs.

4. Service Mesh Mode (Sidecar-Free)

Cilium can provide service mesh features (mTLS, traffic management, L7 observability) without sidecar proxies by using eBPF and an optional Envoy instance per node.

# Enable service mesh during install
cilium install --set kubeProxyReplacement=true \
  --set envoyConfig.enabled=true

# Or via Helm values
helm upgrade cilium cilium/cilium \
  --set l7Proxy=true \
  --set envoyConfig.enabled=true

# Transparent mTLS encryption between pods
# Enable in Cilium ConfigMap or Helm values:
# encryption:
#   enabled: true
#   type: wireguard

# L7 traffic policy with Envoy
apiVersion: cilium.io/v2
kind: CiliumEnvoyConfig
metadata:
  name: rate-limit-api
spec:
  services:
    - name: api-server
      namespace: production
  resources:
    - "@type": type.googleapis.com/envoy.config.listener.v3.Listener
      # Envoy config for rate limiting, header manipulation, etc.

5. eBPF Debugging

# Check Cilium agent status on a node
kubectl exec -n kube-system ds/cilium -- cilium status --verbose

# List endpoints (pods managed by Cilium)
kubectl exec -n kube-system ds/cilium -- cilium endpoint list

# Show BPF maps (connection tracking, policy, NAT)
kubectl exec -n kube-system ds/cilium -- cilium bpf ct list global | head -20
kubectl exec -n kube-system ds/cilium -- cilium bpf policy get --all

# Monitor packet processing in real time
kubectl exec -n kube-system ds/cilium -- cilium monitor
kubectl exec -n kube-system ds/cilium -- cilium monitor --type drop
kubectl exec -n kube-system ds/cilium -- cilium monitor --related-to 12345  # endpoint ID

# Debug specific endpoint
kubectl exec -n kube-system ds/cilium -- cilium endpoint get 12345
kubectl exec -n kube-system ds/cilium -- cilium endpoint log 12345

# Check identity assignment (Cilium assigns numeric identities to pod label sets)
kubectl exec -n kube-system ds/cilium -- cilium identity list

# BPF program status
kubectl exec -n kube-system ds/cilium -- cilium bpf prog list

# Network troubleshooting
kubectl exec -n kube-system ds/cilium -- cilium troubleshoot

6. Cluster Mesh (Multi-Cluster)

# Enable Cluster Mesh
cilium clustermesh enable
cilium clustermesh status

# Connect two clusters
cilium clustermesh connect --destination-context cluster2

# Verify connectivity
cilium clustermesh status --wait

# Global services (accessible across clusters)
# Annotate a service to be shared:
kubectl annotate service my-service \
  service.cilium.io/global="true" \
  service.cilium.io/shared="true"

Debug clue: When Hubble shows DROPPED flows, check hubble observe --verdict DROPPED --namespace production -o json | jq '.flow.drop_reason' for the specific reason code. Common drop reasons: POLICY_DENIED (network policy blocking), CT_MAP_INSERTION_FAILED (conntrack table full — increase with --bpf-ct-global-any-max), and INVALID_PACKET (malformed packet, often a kernel/driver bug).

7. Replacing kube-proxy

# Install Cilium as kube-proxy replacement
cilium install --set kubeProxyReplacement=true

# Verify kube-proxy replacement
kubectl exec -n kube-system ds/cilium -- cilium status | grep KubeProxyReplacement
# Expected: KubeProxyReplacement: True

# Check service handling
kubectl exec -n kube-system ds/cilium -- cilium service list

# After installing, you can remove kube-proxy
kubectl -n kube-system delete ds kube-proxy
kubectl -n kube-system delete cm kube-proxy

Quick Reference

# Installation and status
cilium install --version 1.15.0
cilium status --wait
cilium connectivity test

# Hubble
cilium hubble port-forward &
hubble observe --namespace production
hubble observe --verdict DROPPED
hubble observe --http-status 500

# Debugging
kubectl exec -n kube-system ds/cilium -- cilium status --verbose
kubectl exec -n kube-system ds/cilium -- cilium endpoint list
kubectl exec -n kube-system ds/cilium -- cilium monitor --type drop

# Policy
kubectl get cnp --all-namespaces       # CiliumNetworkPolicies
kubectl get ccnp                        # CiliumClusterwideNetworkPolicies
kubectl get ciliumendpoints -n production

Wiki Navigation

Prerequisites

  • Kubernetes Exercises (Quest Ladder) (CLI) (Exercise Set, L1)

Related Content