Skip to content

Portal | Level: L2: Operations | Topics: Security Scanning | Domain: Security

Security Drills

Remember: The container security checklist: Non-root user, Read-only filesystem, Drop all capabilities, No privilege escalation, Specific image tag (not :latest). Mnemonic: "NRDNS" — "Never Run Dangerous Naked Stuff." Every production container should have all five.

Gotcha: Kubernetes Secrets are base64-encoded, NOT encrypted. Anyone with kubectl get secret -o yaml can decode them with echo <value> | base64 -d. Enable etcd encryption at rest AND restrict RBAC access to secrets. Base64 is encoding, not security.

Default trap: If no NetworkPolicy selects a pod, ALL traffic is allowed by default — ingress and egress. The moment you apply ANY NetworkPolicy that selects a pod, everything not explicitly allowed becomes denied. This "implicit deny on first policy" catches teams off guard and causes outages when they add their first policy.

Drill 1: Container Image Scanning

Difficulty: Easy

Q: Scan a container image for vulnerabilities before deploying it. Show how to integrate into CI.

Answer 
# Trivy scan
trivy image myapp:latest

# Only show HIGH and CRITICAL
trivy image --severity HIGH,CRITICAL myapp:latest

# Fail CI if critical vulns found
trivy image --exit-code 1 --severity CRITICAL myapp:latest

# Scan a Dockerfile for misconfigurations
trivy config Dockerfile

# Scan Kubernetes manifests
trivy config k8s/

# GitHub Actions
- name: Scan image
  uses: aquasecurity/trivy-action@master
  with:
    image-ref: myapp:${{ github.sha }}
    exit-code: 1
    severity: CRITICAL,HIGH

Drill 2: Pod Security Context

Difficulty: Medium

Q: Write a pod spec that follows security best practices: non-root, read-only filesystem, drop all capabilities.

Answer 
apiVersion: v1
kind: Pod
metadata:
  name: secure-app
spec:
  securityContext:
    runAsNonRoot: true
    runAsUser: 1000
    runAsGroup: 1000
    fsGroup: 1000
    seccompProfile:
      type: RuntimeDefault
  containers:
  - name: app
    image: myapp:v1.0
    securityContext:
      allowPrivilegeEscalation: false
      readOnlyRootFilesystem: true
      capabilities:
        drop: ["ALL"]
    volumeMounts:
    - name: tmp
      mountPath: /tmp
  volumes:
  - name: tmp
    emptyDir: {}
Key settings: - `runAsNonRoot: true` — container won't start if image tries to run as root - `readOnlyRootFilesystem: true` — prevents writing to container filesystem - `capabilities.drop: ["ALL"]` — removes all Linux capabilities - `allowPrivilegeEscalation: false` — prevents privilege escalation via setuid Mount `/tmp` as emptyDir for apps that need a writable temp directory.

Drill 3: RBAC

Difficulty: Medium

Q: Create a Role that allows reading pods and logs in the production namespace, and bind it to a ServiceAccount.

Answer 
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: pod-reader
  namespace: production
rules:
- apiGroups: [""]
  resources: ["pods", "pods/log"]
  verbs: ["get", "list", "watch"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: pod-reader-binding
  namespace: production
subjects:
- kind: ServiceAccount
  name: monitoring-sa
  namespace: production
roleRef:
  kind: Role
  name: pod-reader
  apiGroup: rbac.authorization.k8s.io

# Test permissions
kubectl auth can-i get pods -n production --as=system:serviceaccount:production:monitoring-sa
# yes

kubectl auth can-i delete pods -n production --as=system:serviceaccount:production:monitoring-sa
# no

Drill 4: Network Security

Difficulty: Medium

Q: Implement zero-trust networking for a 3-tier app (frontend → api → database).

Answer 
# Default deny all in the namespace
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: default-deny
spec:
  podSelector: {}
  policyTypes: [Ingress, Egress]
---
# Frontend: allow ingress from ingress controller, egress to api
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: frontend-policy
spec:
  podSelector: { matchLabels: { app: frontend } }
  policyTypes: [Ingress, Egress]
  ingress:
  - from:
    - namespaceSelector: { matchLabels: { name: ingress-nginx } }
  egress:
  - to:
    - podSelector: { matchLabels: { app: api } }
    ports: [{ port: 8080 }]
  - to: []  # DNS
    ports: [{ port: 53, protocol: UDP }]
---
# API: allow ingress from frontend, egress to database
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: api-policy
spec:
  podSelector: { matchLabels: { app: api } }
  policyTypes: [Ingress, Egress]
  ingress:
  - from:
    - podSelector: { matchLabels: { app: frontend } }
    ports: [{ port: 8080 }]
  egress:
  - to:
    - podSelector: { matchLabels: { app: database } }
    ports: [{ port: 5432 }]
  - to: []
    ports: [{ port: 53, protocol: UDP }]
---
# Database: allow ingress from api only
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: database-policy
spec:
  podSelector: { matchLabels: { app: database } }
  policyTypes: [Ingress, Egress]
  ingress:
  - from:
    - podSelector: { matchLabels: { app: api } }
    ports: [{ port: 5432 }]

Drill 5: Secret Hygiene

Difficulty: Easy

Q: List 5 things you should NEVER do with secrets in Kubernetes.

Answer 1. **Never commit secrets to Git** — use Sealed Secrets, SOPS, or External Secrets Operator 2. **Never log secrets** — mask in application logs, don't print env vars 3. **Never pass secrets as command-line arguments** — visible in `ps`, process list, `/proc` 4. **Never use default ServiceAccount** — create specific SAs with minimal RBAC 5. **Never store secrets in ConfigMaps** — ConfigMaps aren't encrypted, use Secrets + etcd encryption Also: - Enable etcd encryption at rest - Audit who reads secrets (`kubectl get secrets` in audit logs) - Rotate secrets regularly - Use short-lived credentials (Vault dynamic secrets, IRSA temporary tokens)

Drill 6: Supply Chain Security

Difficulty: Medium

Q: How do you ensure that only verified container images run in your cluster?

Answer 
# 1. Image signing with cosign
cosign sign --key cosign.key registry.example.com/myapp:v1.0

# 2. Verify in admission control (Kyverno)
apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: verify-images
spec:
  validationFailureAction: Enforce
  rules:
  - name: verify-signature
    match:
      any:
      - resources:
          kinds: ["Pod"]
    verifyImages:
    - imageReferences: ["registry.example.com/*"]
      attestors:
      - entries:
        - keys:
            publicKeys: |-
              -----BEGIN PUBLIC KEY-----
              ...
              -----END PUBLIC KEY-----
Full supply chain: 1. **Build**: Reproducible builds, minimal base images (distroless) 2. **Scan**: Trivy/Grype in CI pipeline, fail on critical CVEs 3. **Sign**: cosign signs the image digest 4. **Store**: Private registry with access control 5. **Admit**: Kyverno/Gatekeeper verifies signatures at deploy time 6. **Monitor**: Continuous scanning of running images

Drill 7: Audit Logging

Difficulty: Medium

Q: Enable Kubernetes audit logging to capture who accessed secrets.

Answer 
# /etc/kubernetes/audit-policy.yaml
apiVersion: audit.k8s.io/v1
kind: Policy
rules:
# Log all secret reads at RequestResponse level
- level: RequestResponse
  resources:
  - group: ""
    resources: ["secrets"]
  verbs: ["get", "list"]

# Log all write operations at Request level
- level: Request
  verbs: ["create", "update", "delete", "patch"]

# Log auth failures
- level: Metadata
  users: ["system:anonymous"]

# Don't log read-only health checks
- level: None
  resources:
  - group: ""
    resources: ["endpoints", "services"]
  verbs: ["get", "list", "watch"]

# Add to kube-apiserver flags:
--audit-policy-file=/etc/kubernetes/audit-policy.yaml
--audit-log-path=/var/log/kubernetes/audit.log
--audit-log-maxage=30
--audit-log-maxsize=100
--audit-log-maxbackup=10

# Search audit logs
jq 'select(.objectRef.resource == "secrets")' /var/log/kubernetes/audit.log

Drill 8: Pod-to-Pod Encryption

Difficulty: Easy

Q: How do you encrypt traffic between services in a Kubernetes cluster?

Answer **Option 1: Service Mesh mTLS** (recommended) 
# Istio: automatic mTLS
apiVersion: security.istio.io/v1beta1
kind: PeerAuthentication
metadata:
  name: default
  namespace: production
spec:
  mtls:
    mode: STRICT
All traffic between meshed services is automatically encrypted. **Option 2: Application-level TLS** - Each service has its own cert (cert-manager can automate) - Application code handles TLS - More control, but more work **Option 3: WireGuard/Cilium encryption** 
# Cilium: transparent encryption at the CNI level
cilium encrypt status
# Encrypts all pod-to-pod traffic without application changes
Service mesh is the easiest path — zero application changes, automatic cert rotation.

Drill 9: Vulnerability Response

Difficulty: Hard

Q: A critical CVE is announced for the base image your production services use. What's your response plan?

Answer 
Hour 0-1: Assess
  - What's the CVE? Remote code execution? Data exposure?
  - Which images use the affected base? (check all Dockerfiles)
  - Are running pods vulnerable? (trivy scan deployed images)
  - Is there a public exploit?

Hour 1-4: Mitigate
  - If exploit exists: evaluate if NetworkPolicy/WAF can block the attack vector
  - Rebuild all affected images with patched base
  - Test in staging

Hour 4-8: Patch
  - Deploy patched images to production (rolling update)
  - Verify no regressions
  - Scan all images to confirm patch applied

Day 1-3: Harden
  - Update CI pipeline to fail on this CVE class
  - Add the base image to auto-rebuild triggers
  - Review if the vulnerability could have been caught earlier
  - Postmortem if there was actual impact

# Quick: find all pods using a specific base image
kubectl get pods -A -o json | jq -r \
  '.items[].spec.containers[].image' | sort -u | grep "ubuntu:22"

Drill 10: Least Privilege Checklist

Difficulty: Easy

Q: List the key areas where least privilege should be applied in a Kubernetes cluster.

Answer | Area | Least Privilege Approach | |------|------------------------| | **RBAC** | Specific verbs + resources per SA. No `cluster-admin` for apps. | | **Pod Security** | Non-root, drop capabilities, read-only FS, seccomp | | **Network** | Default deny + explicit allow per service pair | | **Cloud IAM** | IRSA/Workload Identity per SA. No node-wide roles. | | **Secrets** | Encrypt at rest, audit access, rotate regularly | | **Images** | Distroless/minimal base, no package managers in prod | | **Namespaces** | Separate teams/envs, ResourceQuotas, NetworkPolicies | | **API access** | No anonymous auth, restrict API server exposure | | **etcd** | Encrypted, separate certs, not exposed to nodes | | **Registry** | Private registry, signed images, admission control | Test: `kubectl auth can-i --list --as=system:serviceaccount:ns:sa-name`

Wiki Navigation

Prerequisites

Related Content