Environment Variables - Street-Level Ops

Real-world environment variable diagnosis and resolution workflows for production environments.

Task: Debug "command not found" (PATH Issues)

# The symptom: a command you know is installed doesn't work
$ terraform plan
bash: terraform: command not found

# Step 1: Find where the binary actually is
$ find / -name terraform -type f 2>/dev/null
/usr/local/bin/terraform

# Step 2: Check if that directory is in PATH
$ echo $PATH | tr ':' '\n' | grep -n local
1:/usr/local/bin

# If /usr/local/bin IS in PATH but still fails, check for shadowing
$ type -a terraform
# No output = not found anywhere in PATH

# Step 3: Check if you're in the right shell context
$ echo $SHELL     # default shell
$ ps -p $$        # actual current shell

# Step 4: Check for a broken symlink
$ ls -la /usr/local/bin/terraform
lrwxrwxrwx 1 root root 35 Jan 15 10:00 /usr/local/bin/terraform -> /opt/terraform/1.5.0/terraform
$ ls -la /opt/terraform/1.5.0/terraform
ls: cannot access '/opt/terraform/1.5.0/terraform': No such file or directory
# Broken symlink -- someone removed the versioned directory

> **Debug clue:** "command not found" has three common causes in order of likelihood: (1) the directory is not in PATH, (2) the binary is a broken symlink, (3) you are in a different shell than you think (`ps -p $$` shows your actual shell). Check all three before installing anything.

# Fix: Update the symlink to the current version
$ ls /opt/terraform/
1.6.0/
$ sudo ln -sf /opt/terraform/1.6.0/terraform /usr/local/bin/terraform

Under the hood: Environment variables are per-process, stored in the process's memory space and visible at /proc/<pid>/environ. A child process inherits a snapshot of the parent's environment at fork() time. Changing a variable in the parent after forking does not affect the child. This is why "I set it in my shell but the service doesn't see it" is always a process boundary issue.

Task: Find Why an Env Var Is Not Set in systemd

# Service is failing because DATABASE_URL is empty
$ journalctl -u myapp.service | tail
myapp[12345]: Error: DATABASE_URL is not set

# Step 1: Check the unit file
$ systemctl cat myapp.service | grep -i env
Environment="LOG_LEVEL=info"
EnvironmentFile=/etc/myapp/env

# Step 2: Check the environment file
$ cat /etc/myapp/env
LOG_LEVEL=info
REDIS_URL=redis://localhost:6379
# DATABASE_URL is missing from the file

# Step 3: Check if the variable is set elsewhere
$ systemctl show myapp.service --property=Environment
Environment=LOG_LEVEL=info

# The fix: Add the variable to the environment file
$ echo 'DATABASE_URL=postgresql://user:pass@db:5432/app' | sudo tee -a /etc/myapp/env

# Reload and restart
$ sudo systemctl daemon-reload
$ sudo systemctl restart myapp.service

# Verify the process got it
$ PID=$(systemctl show myapp.service --property=MainPID --value)
$ sudo cat /proc/$PID/environ | tr '\0' '\n' | grep DATABASE
DATABASE_URL=postgresql://user:pass@db:5432/app

Task: Inspect Another Process's Environment

# Find the process
$ pgrep -f "java.*myapp"
28491

# Read its environment (requires same user or root)
$ cat /proc/28491/environ | tr '\0' '\n' | sort
DATABASE_URL=postgresql://db:5432/app
HOME=/opt/myapp
JAVA_HOME=/usr/lib/jvm/java-17
LOG_LEVEL=info
PATH=/usr/local/bin:/usr/bin:/bin

# Search for a specific variable
$ cat /proc/28491/environ | tr '\0' '\n' | grep AWS
AWS_REGION=us-east-1
AWS_DEFAULT_REGION=us-east-1

# Compare two processes' environments
$ diff <(cat /proc/28491/environ | tr '\0' '\n' | sort) \
       <(cat /proc/28492/environ | tr '\0' '\n' | sort)

Task: Pass Environment Through sudo

# By default, sudo strips most env vars for security
$ export HTTP_PROXY=http://proxy:3128
$ sudo env | grep HTTP_PROXY
# Nothing -- sudo dropped it

# Option 1: Use sudo -E (preserve entire environment)
$ sudo -E env | grep HTTP_PROXY
HTTP_PROXY=http://proxy:3128
# Requires env_keep or SETENV permission in sudoers

# Option 2: Pass specific variables on the command line
$ sudo HTTP_PROXY=http://proxy:3128 /opt/scripts/deploy.sh

# Option 3: Configure env_keep in sudoers
$ sudo visudo
# Add this line:
# Defaults env_keep += "HTTP_PROXY HTTPS_PROXY NO_PROXY"

# Check what sudo preserves by default
$ sudo sudo -V | grep "Environment variables to preserve"

Gotcha: sudo strips environment variables by default for security (prevents privilege escalation via LD_PRELOAD, PATH manipulation, etc.). This is controlled by env_reset in /etc/sudoers, which is enabled by default. Using sudo -E only works if the sudoers policy allows it — check with sudo -l to see your effective permissions.

Task: Debug Env Vars in SSH Sessions

# Problem: env var is set locally but not available after SSH
# SSH does NOT forward your local environment by default

# Option 1: Set it on the remote in ~/.bashrc or ~/.profile
# (Only works if the shell sources those files -- non-interactive SSH may not)

# Option 2: Send specific variables via SSH config
# In ~/.ssh/config:
# Host prod-*
#     SendEnv DEPLOY_ENV CLUSTER_NAME

# The remote sshd must allow it in /etc/ssh/sshd_config:
# AcceptEnv DEPLOY_ENV CLUSTER_NAME

# Option 3: Pass inline with the command
$ ssh prod-web01 'DEPLOY_ENV=production /opt/scripts/deploy.sh'

# Option 4: Source a remote env file
$ ssh prod-web01 'source /etc/profile.d/deploy.sh && deploy.sh'

# Debug: Check what environment the remote SSH session gets
$ ssh prod-web01 env | sort

Task: Debug Docker Build Args vs Runtime Env

# Dockerfile:
# ARG BUILD_VERSION
# RUN echo "Building $BUILD_VERSION" > /app/version.txt
# ENV APP_VERSION=$BUILD_VERSION

# Build with the arg
$ docker build --build-arg BUILD_VERSION=2.1.0 -t myapp .

# Check what made it into the image environment
$ docker inspect myapp | jq '.[0].Config.Env'
[
  "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
  "APP_VERSION=2.1.0"
]

# ARG values are NOT in the runtime environment unless copied to ENV

> **Remember:** Docker `ARG` = build-time only, `ENV` = runtime persistent. If you need a value at both build and run time, declare both: `ARG VERSION` then `ENV APP_VERSION=$VERSION`. ARGs not copied to ENV vanish after the build stage ends.
$ docker run myapp env | grep BUILD
# Nothing -- ARG is build-time only

$ docker run myapp env | grep APP_VERSION
APP_VERSION=2.1.0
# ENV values persist

Task: Debug Docker Compose Env Var Precedence

# When variables aren't what you expect, trace the precedence

# Check what compose resolves
$ docker compose config | grep -A5 environment

# Check the effective value inside the container
$ docker compose exec web env | grep DATABASE_URL

# Precedence (highest to lowest):
# 1. docker compose run -e DATABASE_URL=... web
# 2. environment: block in docker-compose.yml
# 3. env_file: entries in docker-compose.yml
# 4. ENV in Dockerfile
# 5. .env file in project root (for ${VAR} substitution in compose file itself)

# The .env file in project root is special -- it's used for variable
# substitution in the compose FILE, not injected into containers directly
# To inject into containers, use env_file: directive

Task: Debug Locale Issues (LANG, LC_*)

# Symptom: sorting is wrong, dates look weird, special characters garbled
$ locale
LANG=en_US.UTF-8
LC_COLLATE="en_US.UTF-8"
LC_CTYPE="en_US.UTF-8"
LC_MESSAGES="en_US.UTF-8"
LC_ALL=

# Check available locales
$ locale -a | grep -i utf
en_US.utf8

# If the locale isn't installed
$ sudo locale-gen en_US.UTF-8
$ sudo dpkg-reconfigure locales    # Debian/Ubuntu

# Common fix: force C locale for consistent scripting behavior
$ LC_ALL=C sort myfile.txt

# The LC_ALL variable overrides everything. Use it for scripts
# that need deterministic behavior regardless of user settings.
$ LC_ALL=C grep '[A-Z]' file.txt   # matches only ASCII uppercase
$ grep '[A-Z]' file.txt            # with UTF-8 locale, may match accented chars

# In Docker, locale is often unset. Add to Dockerfile:
# ENV LANG=C.UTF-8
# ENV LC_ALL=C.UTF-8

Task: Debug Env Vars in Cron Jobs

# Step 1: See what cron actually provides
$ crontab -l
* * * * * env > /tmp/cron-env.txt 2>&1

# Wait a minute, then check
$ cat /tmp/cron-env.txt
HOME=/home/deploy
LOGNAME=deploy
PATH=/usr/bin:/bin
SHELL=/bin/sh
# That's it. No custom vars, minimal PATH.

> **Gotcha:** Cron runs with a minimal environment — no `.bashrc`, no `.profile`, no custom PATH. This is the number one reason cron jobs fail silently: the command works in your shell but "command not found" in cron. Always use absolute paths in crontabs or source your environment explicitly.

# Fix option 1: Set vars in crontab
$ crontab -e
PATH=/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin
SHELL=/bin/bash
DATABASE_URL=postgresql://db:5432/app

*/5 * * * * /opt/scripts/backup.sh

# Fix option 2: Source a profile in the job
*/5 * * * * . /etc/profile.d/myapp.sh && /opt/scripts/backup.sh

# Fix option 3: Use a wrapper script that sets up the environment
*/5 * * * * /opt/scripts/run-with-env.sh backup.sh

Task: Audit Environment for Leaked Secrets

# Check if sensitive vars are exposed to child processes
$ env | grep -iE '(password|secret|token|key|credential|api_key)'
AWS_SECRET_ACCESS_KEY=AKIA...
DATABASE_PASSWORD=hunter2

# Check a running process
$ sudo cat /proc/$(pgrep -f myapp)/environ | tr '\0' '\n' | \
  grep -iE '(password|secret|token|key)'

# Check Docker containers
$ docker inspect mycontainer | jq '.[0].Config.Env[]' | \
  grep -iE '(password|secret|token|key)'

# Check if .env files are in git
$ git ls-files | grep -i '\.env'
$ git log --all --diff-filter=A -- '*.env' '.env*'
# Shows if .env files were ever committed (even if later removed)

> **Default trap:** `docker inspect` exposes all environment variables in plaintext, including secrets passed via `-e`. Anyone with Docker socket access can read every secret in every running container. Use Docker secrets or Vault agent injection instead of `-e SECRET=value` for sensitive data.

Pages that link here

• Anti-Primer: Environment Variables
• Environment Variables