Permission Denied

lesson
file-permissions
sudo
capabilities
selinux/apparmor
containers
kubernetes-rbac ---# Permission Denied

Topics: file permissions, sudo, capabilities, SELinux/AppArmor, containers, Kubernetes RBAC Level: L1–L2 (Foundations → Operations) Time: 60–90 minutes Prerequisites: None (everything is explained from scratch)

The Mission¶

$ systemctl restart nginx
Failed to restart nginx.service: Access denied

$ cat /etc/shadow
cat: /etc/shadow: Permission denied

$ docker ps
Got permission denied while trying to connect to the Docker daemon socket

$ kubectl get pods -n production
Error from server (Forbidden): pods is forbidden: User "deploy" cannot list resource "pods"

"Permission denied" is a shape-shifter. It looks like one error, but there are at least seven completely different permission systems that can produce it — and they stack on top of each other. Fixing the wrong layer wastes hours. Fixing it with chmod 777 creates security holes that live forever.

This lesson teaches you to diagnose "permission denied" systematically, the same way a locksmith identifies which lock is jammed rather than breaking down the door.

The Permission Stack¶

Here's the thing most people never learn: Linux has multiple independent permission systems, and access has to pass through ALL of them. Think of it as a series of security checkpoints — even if you clear six of them, the seventh can still deny you.

Layer 1: Unix file permissions (rwx, owner/group/other)
Layer 2: ACLs (Access Control Lists — fine-grained overrides)
Layer 3: sudo / capabilities (privilege escalation)
Layer 4: SELinux / AppArmor (Mandatory Access Control)
Layer 5: Container isolation (namespaces, seccomp)
Layer 6: Kubernetes RBAC (API-level access control)
Layer 7: Cloud IAM (AWS/GCP/Azure — outside the host)

The diagnostic ladder:

Permission Denied
│
├── Is it a file/directory permission? → ls -la, stat
│   └── Wrong owner/group/mode? → chown, chmod
│
├── Are ACLs overriding? → getfacl
│   └── ACL mask limiting effective permissions? → setfacl
│
├── Do you need elevated privileges? → sudo, capabilities
│   └── Missing capability? → setcap, or run through systemd
│
├── Is SELinux/AppArmor blocking? → audit log, getenforce
│   └── Wrong security context? → semanage, restorecon
│
├── Is container isolation blocking? → seccomp, user namespaces
│   └── Running as wrong user? → Dockerfile USER, --user flag
│
├── Is Kubernetes RBAC blocking? → kubectl auth can-i
│   └── Missing Role/RoleBinding? → Create or fix RBAC
│
└── Is cloud IAM blocking? → CloudTrail, policy simulator
    └── Missing IAM permission? → Update policy

Let's work through each layer.

Layer 1: Unix File Permissions¶

The classic. Every file and directory on Linux has: - An owner (one user) - A group (one group) - Permission bits for owner, group, and everyone else

ls -la /etc/shadow
# -rw-r----- 1 root shadow 1.2K Mar 22 10:00 /etc/shadow
#  ↑↑↑↑↑↑↑↑↑   ↑    ↑
#  │││││││││   owner group
#  │││││││││
#  │├─┤├─┤├─┤
#  │ │  │  │
#  │ │  │  └── others: --- (no access)
#  │ │  └── group:  r-- (read only)
#  │ └── owner: rw- (read + write)
#  └── file type: - (regular file)

The meaning of r, w, x changes for directories:

Bit	On a file	On a directory
`r`	Read the file contents	List the directory contents (`ls`)
`w`	Write to the file	Create or delete files in the directory
`x`	Execute the file as a program	Traverse the directory (`cd`)

Gotcha: A directory with w permission lets you delete files inside it even if you don't own those files. This is why /tmp has the sticky bit (t) — it restricts deletion so that only a file's owner (or root) can delete it. Without the sticky bit, any user could delete any other user's temp files.

ls -ld /tmp
# drwxrwxrwt 22 root root 4096 Mar 22 10:00 /tmp
#          ↑
#          └── sticky bit (t instead of x)

The numeric system¶

rwx = 4+2+1 = 7
rw- = 4+2+0 = 6
r-- = 4+0+0 = 4
--- = 0+0+0 = 0

chmod 755 = rwxr-xr-x (owner: full, group: read+execute, others: read+execute)
chmod 644 = rw-r--r-- (owner: read+write, group: read, others: read)
chmod 600 = rw------- (owner: read+write, nobody else)

Special permission bits¶

Bit	Numeric	On files	On directories
SUID	4000	Process runs as file owner, not caller	—
SGID	2000	Process runs with file's group	New files inherit directory's group
Sticky	1000	—	Only file owner can delete

# Find SUID binaries on the system (security audit essential)
find / -perm -4000 -type f 2>/dev/null
# Common: /usr/bin/sudo, /usr/bin/passwd, /usr/bin/ping

Under the Hood: When you run /usr/bin/passwd, the SUID bit means the process runs as root (the file's owner), not as you. That's how a normal user can change their own password — passwd needs root to write to /etc/shadow. Without SUID, every password change would require sudo. The SUID bit is old (1970s) and dangerous — every SUID binary is a potential privilege escalation vector if it has a bug.

War Story: A team ran chmod -R 777 / on a production server to "fix" a permission error on one file. This set the SUID bit on every binary on the system. Every program — including user-installed tools — now ran as root. Every binary became a privilege escalation vector. The fix took two days: reinstall the OS and restore from backup.

The umask¶

When you create a new file, the kernel applies a umask — a mask that removes permission bits from the default. The default for files is 666 (no execute), and for directories is 777.

umask
# → 0022

# 666 - 022 = 644 → new files are rw-r--r--
# 777 - 022 = 755 → new directories are rwxr-xr-x

Gotcha: umask 000 means new files are world-readable and world-writable. If you're writing config files with credentials, this leaks them to every user on the system. Some applications set their own umask, but many inherit yours.

Layer 2: ACLs — When Owner/Group/Other Isn't Enough¶

Traditional Unix permissions give you one owner and one group. ACLs let you grant access to specific additional users or groups:

# View ACLs
getfacl /var/www/html/
# → user::rwx
#    user:deploy:rwx        ← specific user ACL
#    group::r-x
#    group:devteam:rwx      ← specific group ACL
#    mask::rwx
#    other::r-x

# Grant access to a specific user
setfacl -m u:deploy:rwx /var/www/html/

# Grant access to a specific group
setfacl -m g:devteam:rwx /var/www/html/

Gotcha: ACLs have a mask that acts as a ceiling on effective permissions. Here's the trap: running chmod on a file with ACLs modifies the mask, not the group permissions. So chmod 750 on a file with ACL user:deploy:rwx silently reduces deploy's effective permissions to r-x because the mask becomes r-x. This is the most subtle permission bug on Linux — the ACL says rwx, but the effective permission is r-x.

# Check effective permissions (shows mask interaction)
getfacl /var/www/html/file
# → user:deploy:rwx     #effective:r-x
#                         ↑ mask is limiting this

Layer 3: Privileges — sudo and Capabilities¶

Some operations require elevated privileges. You have two options: the old way (sudo/SUID) and the modern way (capabilities).

sudo¶

# Check if you can run a command
sudo -l
# → (ALL) NOPASSWD: /usr/bin/systemctl restart nginx
# → (ALL) NOPASSWD: /usr/bin/docker *

Gotcha: Editing /etc/sudoers with vim instead of visudo is playing Russian roulette. visudo validates syntax before saving. A syntax error in sudoers breaks sudo for everyone — including root if you're relying on sudo to become root. Recovery requires single-user mode or a live boot. Always use visudo or drop files in /etc/sudoers.d/ (but note: filenames with . or ~ in them are silently ignored).

Trivia: sudo was written by Bob Coggeshall and Cliff Spencer at SUNY Buffalo around 1980 because they kept forgetting the root password. It logs every invocation to syslog: Mar 15 10:45:22 webserver sudo: deploy : TTY=pts/0 ; USER=root ; COMMAND=/usr/bin/systemctl restart nginx

Capabilities — the modern alternative¶

Root is all-or-nothing. Capabilities split root's power into 41 individual privileges:

Capability	What it allows
`CAP_NET_BIND_SERVICE`	Bind to ports below 1024
`CAP_NET_RAW`	Raw sockets (ping, tcpdump)
`CAP_SYS_PTRACE`	Trace processes (strace, gdb)
`CAP_SYS_ADMIN`	Mount, load modules, etc. — effectively root
`CAP_DAC_OVERRIDE`	Bypass file permission checks entirely

# Give a binary a specific capability instead of SUID
sudo setcap 'cap_net_bind_service=+ep' /usr/bin/myapp

# Check capabilities on a binary
getcap /usr/bin/myapp
# → /usr/bin/myapp cap_net_bind_service=ep

# Check capabilities of a running process
cat /proc/<pid>/status | grep Cap

Gotcha: CAP_SYS_ADMIN is the junk drawer of capabilities. It covers so many operations (mount, chroot, load kernel modules, set hostname, modify cgroups...) that granting it is essentially granting root. Whenever possible, use a more specific capability.

Layer 4: SELinux and AppArmor — Mandatory Access Control¶

This is where most people hit a wall. Even with correct file permissions, even running as root, SELinux or AppArmor can still deny you.

The mental model¶

Traditional permissions (Layers 1-3) are Discretionary Access Control (DAC) — the file owner decides who can access it, and root bypasses everything.

SELinux/AppArmor add Mandatory Access Control (MAC) — a second, independent gate that even root can't bypass. Both gates must pass:

DAC check: Does the user have rwx permission? → Yes
MAC check: Does the security policy allow this operation? → DENIED
Result: Permission denied (even though rwx said yes)

SELinux (RHEL, CentOS, Fedora)¶

# Is SELinux the problem?
getenforce
# → Enforcing  (blocking violations)
# → Permissive (logging but not blocking)
# → Disabled

# Check the security context on a file
ls -Z /var/www/html/
# → system_u:object_r:httpd_sys_content_t:s0 index.html
#                       ↑
#                       This is the "type" — 95% of policy decisions use this

# Check recent denials
ausearch -m AVC -ts recent
# → type=AVC msg=audit(...): avc: denied { read } for name="index.html"
#   scontext=system_u:system_r:httpd_t:s0
#   tcontext=system_u:object_r:default_t:s0

The most common SELinux problem: you copied or moved files, and they have the wrong type label.

# WRONG — mv preserves the source context
mv ~/website/* /var/www/html/
# Files keep home_dir_t context → Apache can't read them

# Fix — restore the correct labels
restorecon -Rv /var/www/html/
# → Relabeled /var/www/html/index.html from home_dir_t to httpd_sys_content_t

Gotcha: setenforce 0 (turning off SELinux) is NOT a fix. It's disabling your mandatory access control because you didn't want to figure out the correct label. Use audit2why to understand what's being denied, then fix the label or boolean.

# Understand WHY something was denied
ausearch -m AVC -ts recent | audit2why
# → "Was caused by: Missing type enforcement (TE) allow rule"
# → "Possible fix: setsebool -P httpd_can_network_connect on"

Name Origin: SELinux was developed by the NSA (yes, the National Security Agency) and released as open source in 2000, merged into the kernel in 2.6 (2003). The NSA's involvement made many people suspicious, but the code is fully open and has been audited extensively. AppArmor (Ubuntu, SUSE, Debian) is the path-based alternative — simpler to learn but less powerful.

The chcon vs semanage trap¶

# Temporary — lost on relabel or restorecon
chcon -t httpd_sys_content_t /srv/www/index.html

# Permanent — survives restorecon, reboots, everything
semanage fcontext -a -t httpd_sys_content_t "/srv/www(/.*)?"
restorecon -Rv /srv/www/

Remember: "chcon is a Cheat, semanage is the Seal." chcon is temporary (useful for testing), semanage is permanent (use for production).

Layer 5: Container Isolation¶

Containers add their own permission layers. Even if the host permissions are fine, the container's isolation can deny access.

Running as the wrong user¶

# BAD — running as root inside the container
FROM python:3.11
COPY app.py /app/
CMD ["python", "/app/app.py"]

# GOOD — non-root user
FROM python:3.11
RUN useradd -r -u 1000 appuser
COPY --chown=appuser:appuser app.py /app/
USER appuser
CMD ["python", "/app/app.py"]

Volume mount permissions¶

# Host file owned by user 1000
ls -la /data/config.yaml
# → -rw-r----- 1 deploy deploy 512 Mar 22 /data/config.yaml

# Container runs as user 999 (defined in Dockerfile)
docker run -v /data:/data myimage cat /data/config.yaml
# → Permission denied (UID 999 can't read a file owned by UID 1000)

The fix: align UIDs between host and container, or use named volumes that Docker manages.

Dropped capabilities¶

# Docker drops many capabilities by default
docker run --rm alpine cat /proc/1/status | grep Cap
# The container can't do things like raw sockets, loading modules, etc.

# If your app needs a specific capability:
docker run --cap-add=NET_RAW myimage ping google.com

SELinux and volume mounts¶

On SELinux-enabled hosts, volume mounts need a special suffix:

# Without :Z, SELinux blocks the container from reading the volume
docker run -v /data:/data myimage   # → Permission denied inside container

# :Z = private label (only this container can access)
docker run -v /data:/data:Z myimage  # → Works

# :z = shared label (multiple containers can access)
docker run -v /data:/data:z myimage  # → Works, shared

Gotcha: --privileged disables most container security — all capabilities, no seccomp, full device access. It's the container equivalent of chmod 777. Never use it in production unless you have a documented, reviewed exception.

Layer 6: Kubernetes RBAC¶

In Kubernetes, "permission denied" comes from the API server's Role-Based Access Control:

$ kubectl get pods -n production
Error from server (Forbidden): pods is forbidden: User "deploy" cannot list resource "pods"
in API group "" in the namespace "production"

RBAC has four objects — the 2R-2B model:

Object	Scope	Purpose
Role	Namespace	Defines what actions are allowed
ClusterRole	Cluster-wide	Same, but for cluster-scoped resources or reusable across namespaces
RoleBinding	Namespace	Grants a Role (or ClusterRole) to a user/group/SA in one namespace
ClusterRoleBinding	Cluster-wide	Grants a ClusterRole to a user/group/SA everywhere

The binding type determines scope, NOT the role type. A ClusterRole + RoleBinding = scoped to one namespace.

Diagnosing RBAC¶

# Can I do this?
kubectl auth can-i create deployments -n production
# → no

# Can a specific service account do this?
kubectl auth can-i list pods -n staging \
  --as=system:serviceaccount:ci:ci-deployer
# → yes

# List all my permissions in a namespace
kubectl auth can-i --list -n production

Common RBAC mistakes¶

# MISTAKE: ClusterRoleBinding for a staging deployer
# This SA can now deploy to EVERY namespace, including production
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding  # ← Should be RoleBinding
...

# FIX: Use RoleBinding scoped to staging only
kind: RoleBinding
metadata:
  namespace: staging  # ← scoped

Gotcha: pods and pods/log are different resources in RBAC. A role that allows get pods does NOT allow get pods/log. Same for pods/exec and pods/portforward — they're all separate. This trips up everyone writing their first custom role.

War Story: Tesla's Kubernetes cluster (2018) had an unauthenticated dashboard with cluster-admin access. Attackers deployed cryptocurrency miners and accessed S3 buckets containing telemetry data. The fix was trivial — RBAC — but the damage was done. Always audit: does every ServiceAccount actually need the permissions it has?

Layer 7: Cloud IAM¶

If you're on AWS, GCP, or Azure, there's a permission system outside the host entirely:

# AWS: "Why can't my EC2 instance access S3?"
aws sts get-caller-identity
# → Check which role the instance is using

aws s3 ls s3://my-bucket/
# → An error occurred (AccessDenied)
# Check: IAM role policy, S3 bucket policy, S3 ACL — any can deny

Cloud IAM is invisible from inside the instance — ls -la, sudo, SELinux all look fine.

War Story: A cloud platform engineer scoped down a CI/CD IAM role from s3:* to specific bucket ARNs. The policy used a pattern app-services/*-service but one team's repo was named payment-gateway — didn't match the pattern. Deploy broke. The engineer tried to fix the policy, but the CI/CD role couldn't modify its own IAM policy (that permission was also removed). Break-glass credentials had expired 8 months ago. Root account MFA was linked to the former CTO's old phone. Recovery took two days and a phone call to the ex-CTO.

The Complete Decision Tree¶

Permission Denied
│
├── File/directory? → ls -la, stat
│   ├── Wrong owner → chown user:group path
│   ├── Wrong mode → chmod 644 path (or appropriate)
│   └── Need sticky/SGID? → chmod +t dir, chmod g+s dir
│
├── ACL override? → getfacl path
│   └── Mask limiting? → setfacl -m m::rwx path
│
├── Need privileges? → sudo -l
│   ├── Need sudo → visudo, add to sudoers.d/
│   ├── Need specific capability → setcap, or systemd AmbientCapabilities=
│   └── Port < 1024 → CAP_NET_BIND_SERVICE or reverse proxy
│
├── SELinux/AppArmor? → getenforce, ausearch -m AVC
│   ├── Wrong file context → restorecon -Rv path
│   ├── Need boolean → setsebool -P boolean_name on
│   └── Need port → semanage port -a -t type -p tcp PORT
│
├── Container? → docker inspect, seccomp profile
│   ├── Wrong UID → USER in Dockerfile or --user flag
│   ├── Volume mount → Align UIDs or use :Z/:z on SELinux
│   └── Need capability → --cap-add=CAPABILITY
│
├── Kubernetes? → kubectl auth can-i
│   ├── Missing Role → Create Role with needed verbs/resources
│   ├── Missing Binding → Create RoleBinding (not ClusterRoleBinding!)
│   └── Subresource? → Check pods/log, pods/exec separately
│
└── Cloud IAM? → CloudTrail, IAM policy simulator
    └── Missing permission → Update policy (but who has permission to update policies?)

Flashcard Check¶

Q1: drwxrwxrwt — what does the t mean?

Sticky bit. On directories, only the file owner (or root) can delete files inside. Used on /tmp to prevent users from deleting each other's temp files.

Q2: You set an ACL user:deploy:rwx but deploy can only read. Why?

The ACL mask is limiting effective permissions. chmod on a file with ACLs modifies the mask. Check with getfacl — look for #effective:r-x next to the ACL entry.

Q3: What's the difference between chcon and semanage fcontext?

chcon is temporary — labels are lost on restorecon or relabel. semanage creates a permanent rule in the policy database. Use semanage for production.

Q4: kubectl auth can-i get pods returns yes, but kubectl logs <pod> is forbidden. Why?

pods and pods/log are separate resources in RBAC. The role needs explicit permission for the pods/log subresource.

Q5: Docker volume mount is "permission denied" on a SELinux host. Quick fix?

Add :Z (private) or :z (shared) to the volume mount: -v /data:/data:Z

Q6: New files are created with 0666 permissions even though you expect 0644. Why?

Your umask is 0000 instead of 0022. Check with umask. Fix in ~/.bashrc or the service's systemd unit.

Q7: CAP_SYS_ADMIN — should you grant it?

Almost never. It covers so many operations it's effectively root. Find the specific capability you actually need (NET_BIND_SERVICE, NET_RAW, etc.).

Q8: Why should you NEVER edit /etc/sudoers with vim?

A syntax error breaks sudo for everyone. visudo validates syntax before saving. Recovery from a broken sudoers requires single-user mode or live boot.

Exercises¶

Exercise 1: Decode the permissions (think)¶

For each ls -la line, answer: can user deploy (in group www-data) read the file?

-rw-r----- 1 root   shadow   1.2K  /etc/shadow
-rwxr-x--- 1 root   www-data 4.0K  /var/www/html/index.html
-rw------- 1 deploy deploy   512   /home/deploy/.ssh/id_rsa
drwxr-x--T 2 root   www-data 4096  /var/www/uploads/

Answers

1. `/etc/shadow` — **No.** Owner=root (rw-), group=shadow (r--), others=(---). deploy is not root and not in group shadow. 2. `/var/www/html/index.html` — **Yes.** deploy is in group www-data, which has r-x. Can read and execute, but not write. 3. `/home/deploy/.ssh/id_rsa` — **Yes.** Owner=deploy (rw-). This is the correct permission for SSH private keys — 600. 4. `/var/www/uploads/` — **Yes, can enter and list** (group www-data has r-x). The `T` (sticky bit + no others-execute) means only file owners can delete their own files inside.

Exercise 2: Fix the SELinux denial (hands-on, RHEL/CentOS/Fedora)¶

If you have access to an SELinux system:

# Create a file in home directory and move it to the web root
echo "hello" > ~/test.html
sudo mv ~/test.html /var/www/html/

# Check what Apache sees
curl http://localhost/test.html
# → 403 Forbidden

# Diagnose
ls -Z /var/www/html/test.html

Why is Apache denied? Fix it two ways — the temporary way and the permanent way.

Solution

# The file has the wrong SELinux context (home_dir_t instead of httpd_sys_content_t)
ls -Z /var/www/html/test.html
# → unconfined_u:object_r:user_home_t:s0 test.html

# Temporary fix
chcon -t httpd_sys_content_t /var/www/html/test.html

# Permanent fix (better — use this in production)
restorecon -v /var/www/html/test.html
# This works because /var/www/html already has a default context defined

# Verify
curl http://localhost/test.html
# → hello

The root cause: `mv` preserves the source file's security context. `cp` creates a new file with the destination directory's default context. On SELinux systems, always prefer `cp` over `mv` into web roots, or run `restorecon` after moving.

Exercise 3: RBAC debugging (Kubernetes)¶

Your CI/CD pipeline runs in the ci namespace using service account ci-deployer. It can create deployments in staging but gets Forbidden in production. Write the kubectl commands to diagnose this, and then write the RBAC objects to fix it.

Diagnostic commands

# Check what the SA can do in staging
kubectl auth can-i --list -n staging \
  --as=system:serviceaccount:ci:ci-deployer

# Check what it can do in production
kubectl auth can-i --list -n production \
  --as=system:serviceaccount:ci:ci-deployer

# Check what bindings exist
kubectl get rolebinding -n staging -o yaml
kubectl get rolebinding -n production -o yaml

Fix (RBAC objects)

# The Role probably already exists (reuse ClusterRole if you have one)
# Just add a RoleBinding in the production namespace:
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: ci-deployer-binding
  namespace: production    # ← scoped to production
subjects:
  - kind: ServiceAccount
    name: ci-deployer
    namespace: ci
roleRef:
  kind: ClusterRole
  name: deployer           # ← reuse existing ClusterRole
  apiGroup: rbac.authorization.k8s.io

Use `RoleBinding`, not `ClusterRoleBinding` — scope the permission to the exact namespace needed.

Exercise 4: The decision (think, don't code)¶

For each scenario, identify the permission layer and the first command to run:

A web app can't read uploaded files in /var/www/uploads/
A Docker container can't write to a mounted volume
ping stopped working for non-root users after a system update
A Kubernetes pod can't connect to the API server
An EC2 instance can't read from an S3 bucket

Answers

1. **Layer 1 or 4.** First: `ls -la /var/www/uploads/` — check if web server user has read permission. If permissions look correct, check SELinux: `ls -Z /var/www/uploads/`. Files uploaded via a web form may get the wrong context. 2. **Layer 5 (container).** First: `docker exec mycontainer id` — what UID is the process? Then `ls -la` on the host to check if that UID has access. Also check SELinux `:Z`/`:z` suffix on SELinux hosts. 3. **Layer 3 (capabilities).** `ping` uses raw sockets (CAP_NET_RAW). If the update removed the SUID bit or capability, non-root users lose ping. Check: `getcap /usr/bin/ping` or `ls -la /usr/bin/ping` (look for `s` in permissions). 4. **Layer 6 (RBAC).** First: `kubectl auth can-i --list` from inside the pod. Check if `automountServiceAccountToken` is false (no token mounted = no API access). Check the ServiceAccount's RoleBindings. 5. **Layer 7 (Cloud IAM).** First: `aws sts get-caller-identity` to confirm which IAM role. Then check the role's policy, the S3 bucket policy, and any S3 ACLs. Use the IAM Policy Simulator for complex cases.

Cheat Sheet¶

Quick Diagnosis¶

Layer	Check command	What it tells you
File permissions	`ls -la path`	Owner, group, mode bits
ACLs	`getfacl path`	Extended permissions, mask
sudo	`sudo -l`	What you can run elevated
Capabilities	`getcap /usr/bin/binary`	Specific privileges granted
SELinux	`getenforce` + `ls -Z path`	MAC enforcement, security context
AppArmor	`aa-status`	Profile enforcement status
Container	`docker exec ctr id`	UID inside container
K8s RBAC	`kubectl auth can-i verb resource`	API permission check
Cloud IAM	`aws sts get-caller-identity`	Active role/user

Permission Cheat Codes¶

Common need	Command
Make file readable by group	`chmod g+r file`
Set owner recursively	`chown -R user:group dir/`
Fix SELinux context	`restorecon -Rv /path/`
Check SELinux denials	`ausearch -m AVC -ts recent`
Add Linux capability	`setcap 'cap_net_bind_service=+ep' binary`
Docker volume on SELinux	`-v /data:/data:Z`
K8s permission check	`kubectl auth can-i --list -n namespace`

SSH Key Permissions (must be exact)¶

~/.ssh/             → 700
~/.ssh/id_rsa       → 600
~/.ssh/id_rsa.pub   → 644
~/.ssh/authorized_keys → 600
~/.ssh/config       → 644

SSH silently refuses keys with wrong permissions. No error in the client — just "Permission denied (publickey)."

Takeaways¶

Permission systems stack. File permissions → ACLs → sudo/capabilities → SELinux → container isolation → RBAC → cloud IAM. Access must pass ALL layers.
chmod 777 is never the answer. It doesn't fix the real problem (wrong owner, wrong SELinux context, wrong capability) and creates security holes that persist forever.
SELinux denials are not mysterious. ausearch -m AVC tells you exactly what was denied and why. audit2why suggests the fix. Don't disable SELinux — learn to read the audit log.
Container UID alignment matters. The UID inside the container must match the UID that owns the mounted volume on the host. Docker doesn't translate UIDs by default.
RBAC binding type determines scope. ClusterRoleBinding grants permissions everywhere. RoleBinding scopes to one namespace. Almost always use RoleBinding.
Cloud IAM is invisible from inside. If everything on the host looks correct, check the cloud layer — security groups, IAM roles, bucket policies.

Connection Refused — diagnostic ladder for network access failures
The Hanging Deploy — when processes don't respond to signals

Permission Denied

The Mission¶

The Permission Stack¶

Layer 1: Unix File Permissions¶

The numeric system¶

Special permission bits¶

The umask¶

Layer 2: ACLs — When Owner/Group/Other Isn't Enough¶

Layer 3: Privileges — sudo and Capabilities¶

sudo¶

Capabilities — the modern alternative¶

Layer 4: SELinux and AppArmor — Mandatory Access Control¶

The mental model¶

SELinux (RHEL, CentOS, Fedora)¶

The chcon vs semanage trap¶

Layer 5: Container Isolation¶

Running as the wrong user¶

Volume mount permissions¶

Dropped capabilities¶

SELinux and volume mounts¶

Layer 6: Kubernetes RBAC¶

Diagnosing RBAC¶

Common RBAC mistakes¶

Layer 7: Cloud IAM¶

The Complete Decision Tree¶

Flashcard Check¶

Exercises¶

Exercise 1: Decode the permissions (think)¶

Exercise 2: Fix the SELinux denial (hands-on, RHEL/CentOS/Fedora)¶

Exercise 3: RBAC debugging (Kubernetes)¶

Exercise 4: The decision (think, don't code)¶

Cheat Sheet¶

Quick Diagnosis¶

Permission Cheat Codes¶

SSH Key Permissions (must be exact)¶

Takeaways¶

Related Lessons¶

Pages that link here¶