Skip to content

Portal | Level: L2: Operations | Topics: RBAC | Domain: Kubernetes

Runbook: RBAC Forbidden Error

Symptoms

  • API calls return 403 Forbidden or Error from server (Forbidden)
  • Service account can't perform expected operations
  • Deployment or CI job fails with permission error

Fast Triage

# Check what the service account can do
kubectl auth can-i --list --as=system:serviceaccount:grokdevops:default -n grokdevops

# Check specific permission
kubectl auth can-i get pods --as=system:serviceaccount:grokdevops:default -n grokdevops

# List roles and bindings
kubectl get roles,rolebindings -n grokdevops
kubectl get clusterroles,clusterrolebindings | grep grokdevops

# Check the service account
kubectl get serviceaccount -n grokdevops

Likely Causes (ranked)

  1. Missing RoleBinding — Role exists but not bound to the service account
  2. Wrong namespace — RoleBinding in different namespace than the resource
  3. ClusterRole needed — operation requires cluster-wide permission
  4. Service account mismatch — pod using wrong service account

Evidence Interpretation

What bad looks like: 

Error from server (Forbidden): pods is forbidden: User "system:serviceaccount:grokdevops:default"
cannot list resource "pods" in API group "" in the namespace "grokdevops"
- The error message tells you exactly what is missing: the verb (list), the resource (pods), the API group (""), and the namespace (grokdevops). - Parse the service account name: system:serviceaccount:<namespace>:<name> — verify the pod is using the expected service account. - kubectl auth can-i with --as flag lets you test permissions without deploying anything.

Fix Steps

  1. Identify what permission is needed from the error message
  2. Create or fix RoleBinding: 
    kubectl create rolebinding <name> \
  --role=<role-name> \
  --serviceaccount=grokdevops:default \
  -n grokdevops
  3. For cluster-wide permissions: 
    kubectl create clusterrolebinding <name> \
  --clusterrole=<role-name> \
  --serviceaccount=grokdevops:default

Verification

kubectl auth can-i <verb> <resource> --as=system:serviceaccount:grokdevops:default -n grokdevops

Cleanup

Remove any overly permissive bindings created during debugging: 

kubectl delete rolebinding <temp-binding> -n grokdevops

Unknown Unknowns

  • RBAC is deny-by-default. If no Role/ClusterRole grants a permission, it is denied — there is no "implicit allow."
  • A Role is namespaced (only grants access within one namespace). A ClusterRole can grant cluster-wide access or be reused across namespaces via RoleBindings.
  • A RoleBinding can reference a ClusterRole — this grants the ClusterRole's permissions but scoped to the RoleBinding's namespace. This is a common and intentional pattern.
  • Aggregated ClusterRoles (using aggregationRule) automatically combine permissions from other ClusterRoles matching certain labels. Editing them directly gets overwritten.

Pitfalls

  • Creating overly permissive bindings — granting cluster-admin to fix a 403 works but violates least privilege. Identify the exact verb and resource needed.
  • Forgetting to specify namespace — a RoleBinding in namespace default does not grant access to namespace grokdevops. Always check the namespace of both the binding and the resource.
  • Confusing Role with ClusterRole scope — a Role only works in its namespace. If the workload needs cross-namespace or cluster-scoped resources (nodes, PVs), you need a ClusterRole + ClusterRoleBinding.

See Also

  • training/interactive/exercises/levels/level-41/k8s-rbac-permission/
  • training/interview-scenarios/09-rbac-forbidden.md
  • training/library/guides/troubleshooting.md

Wiki Navigation

Related Content