Environment Variables Footguns¶

Mistakes that cause broken deployments, security exposure, or hours of debugging.

1. LD_LIBRARY_PATH breaking system tools¶

You set LD_LIBRARY_PATH=/opt/myapp/lib globally (in /etc/profile.d/ or exported in .bashrc) because your application needs a custom library. Now every binary on the system searches that directory first for shared libraries. If your app ships a different version of libssl, libc, or libz, system tools like ssh, curl, apt, and even sudo start segfaulting or behaving unpredictably.

Fix: Never set LD_LIBRARY_PATH globally. Use it only in the wrapper script for the specific application, or better, set the RPATH at compile time so the binary knows where its libraries are without affecting the global search path.

2. PATH order hijacking (first match wins)¶

PATH is searched left to right. The first matching executable wins. If you prepend a directory to PATH and that directory contains a binary with the same name as a system tool, the system tool is silently replaced.

# Dangerous: someone drops a custom 'curl' in /opt/tools/bin
export PATH="/opt/tools/bin:$PATH"
# Now every 'curl' invocation runs the custom one, not /usr/bin/curl
# If this is a wrapper that calls curl, you get infinite recursion

Fix: Always check for shadowing after modifying PATH: type -a curl. Put custom paths at the end of PATH unless you explicitly need to override system tools, and document it.

3. Environment variables with spaces and special characters¶

Variables containing spaces, quotes, or shell metacharacters cause silent failures when used unquoted:

export DEPLOY_MSG="deploy to prod (v2.1)"
# Works fine here, but...

# In a poorly written script:
curl -X POST -d message=$DEPLOY_MSG http://slack/webhook
# Shell splits on spaces, parentheses cause syntax issues

# In a systemd unit:
Environment="DEPLOY_MSG=deploy to prod (v2.1)"
# The quotes around the whole assignment are required in systemd
# Environment=DEPLOY_MSG=deploy to prod    <-- WRONG: "to" and "prod" are separate assignments

Fix: Always double-quote variable expansions in scripts: "$DEPLOY_MSG". In systemd, wrap the entire KEY=VALUE in quotes.

4. export does not persist across sessions¶

New engineers set a variable with export and expect it to survive a reboot or new terminal. It does not. export modifies the current process's environment, which is inherited by children but lost when the shell exits.

export DATABASE_URL="postgresql://db:5432/app"
# Log out, log back in:
echo $DATABASE_URL
# Empty

Fix: Put persistent variables in the appropriate shell initialization file (~/.bash_profile for login shells, /etc/profile.d/ for system-wide). For services, use systemd EnvironmentFile=.

5. .env files committed to git (secrets leak)¶

Someone adds database credentials or API keys to .env, forgets to add .env to .gitignore, and pushes. Even if you later remove the file and add it to .gitignore, the credentials remain in git history forever.

# Check if it's already been committed
$ git log --all --diff-filter=A -- '.env' '.env.*'
commit abc123  Author: dev@company.com  Date: 6 months ago
    initial commit

# The secret has been in every clone for 6 months

Fix: Add .env to .gitignore before the first commit. Provide a .env.example with placeholder values. If secrets were committed, rotate them immediately -- removing the file from git history (BFG, git-filter-repo) is good hygiene but does not protect against anyone who already cloned.

6. ENV in Dockerfile RUN vs runtime confusion¶

ARG values are available during build. ENV values persist in the image. People confuse them and expose build secrets in the runtime image:

# WRONG: This embeds the secret in the image metadata forever
ARG DB_PASSWORD
ENV DB_PASSWORD=$DB_PASSWORD
RUN echo "Connecting to DB..." && ./migrate.sh

# docker inspect shows: "DB_PASSWORD=hunter2"
# Anyone with image access can read it

Fix: For build-time secrets, use ARG only (it won't persist in the image) or use BuildKit's --mount=type=secret. For runtime config, inject via docker run --env or --env-file, not baked into the image.

Gotcha: Even ARG values are not truly secret during build. They appear in the image layer history (docker history --no-trunc <image>) and in the build cache. BuildKit's --mount=type=secret is the only Docker-native method that keeps secrets out of the image layers entirely. It mounts the secret as a tmpfs file visible only during that single RUN instruction.

7. sudo stripping environment variables¶

By default, sudo resets the environment to a safe set defined by env_reset in /etc/sudoers. Your carefully set http_proxy, KUBECONFIG, AWS_PROFILE, and other variables disappear.

$ export KUBECONFIG=/home/deploy/.kube/prod-config
$ sudo kubectl get nodes
# Uses default kubeconfig (/root/.kube/config), not yours

Fix: Use sudo -E (if permitted), pass variables explicitly (sudo KUBECONFIG=$KUBECONFIG kubectl ...), or configure env_keep in sudoers for specific variables.

8. Cron's minimal PATH causes silent failures¶

Cron jobs run with PATH=/usr/bin:/bin (varies by distro). Any command in /usr/local/bin, /snap/bin, /opt/, or $HOME/bin will fail with "command not found". The error goes to cron's mail system, which nobody reads.

# Works in your shell:
$ which node
/usr/local/bin/node

# In cron: "node: command not found"
# Because /usr/local/bin is not in cron's PATH

Fix: Always set PATH explicitly at the top of your crontab, or use absolute paths in cron commands. Redirect stderr to a log file so failures are visible: */5 * * * * /opt/scripts/job.sh >> /var/log/myjob.log 2>&1

9. Environment variables with newlines or null bytes¶

Some tools truncate or mangle variables containing newlines. Kubernetes secrets mounted as env vars, multiline certificates, and JSON blobs stored in env vars are common culprits:

# Looks fine in your terminal:
export CERT="-----BEGIN CERTIFICATE-----
MIIBxTCCAW...
-----END CERTIFICATE-----"

# But in a Docker --env-file, newlines are not supported
# The file format is KEY=VALUE per line -- the cert gets truncated at the first newline

# In Kubernetes env vars, newlines work in the YAML but may break
# applications that read the var incorrectly

Fix: For multiline values, use file-based injection (mount as a volume, use Kubernetes secrets as files, use Docker secrets). Base64-encode if you must use an env var, and decode in the application.

10. Overriding system env vars accidentally¶

Setting HOME, USER, SHELL, or TERM to incorrect values causes subtle, hard-to-trace breakage:

# In a script that processes user data:
export HOME=/tmp/workdir
# Now git, SSH, GPG, and any tool that reads ~/.config/* are broken
# SSH can't find ~/.ssh/known_hosts
# Git can't find ~/.gitconfig

# In a Dockerfile:
ENV HOME=/app
# Tools expecting /root or /home/appuser will look in the wrong place

Fix: Never reassign system variables (HOME, USER, SHELL, PATH, TERM, LANG) unless you fully understand every downstream consumer. Use application-specific variable names (APP_HOME, WORK_DIR) instead.