Skip to content

Package Management - Street-Level Ops

Real-world package management workflows for production Linux servers.

Triage: "What version is running and where did it come from?"

# Debian/Ubuntu
dpkg -l nginx
# ii  nginx  1.24.0-1ubuntu1  amd64  high performance web server

apt-cache policy nginx
# nginx:
#   Installed: 1.24.0-1ubuntu1
#   Candidate: 1.24.0-2ubuntu1
#   Version table:
#      1.24.0-2ubuntu1 500
#         500 http://archive.ubuntu.com/ubuntu noble/main amd64 Packages
#   *** 1.24.0-1ubuntu1 100

# RHEL/CentOS
rpm -qi nginx | grep -E "Name|Version|Release|Repo"
# Name        : nginx
# Version     : 1.24.0
# Release     : 1.el9
dnf info --installed nginx

Find what package owns a file

# Debian — "what dropped this binary?"
dpkg -S /usr/sbin/nginx
# nginx-core: /usr/sbin/nginx

# RHEL
rpm -qf /usr/sbin/nginx
# nginx-1.24.0-1.el9.x86_64

# File not from a package? Check if it was manually placed
dpkg -S /opt/custom-agent/bin/agent 2>&1
# dpkg-query: no path found matching pattern /opt/custom-agent/bin/agent

Emergency: Fix broken dpkg state at 3 AM

# Step 1 — what is broken?
dpkg --audit
# The following packages are in a half-configured state:
#  libssl3:amd64 3.0.13-0ubuntu3

# Step 2 — let apt try to fix dependencies
apt --fix-broken install

# Step 3 — finish stalled configurations
dpkg --configure -a

# Step 4 — nuclear option if package is wedged (data loss possible)
dpkg --remove --force-remove-reinstreq libssl3
apt install libssl3

# Step 5 — check the logs to understand what happened
tail -30 /var/log/dpkg.log
grep -i error /var/log/apt/term.log | tail -20

Debug clue: When dpkg --audit shows half-configured packages, the root cause is almost always a failed postinst script. Check /var/lib/dpkg/info/<package>.postinst to see what it does, then look at /var/log/dpkg.log for the exact error. Sometimes manually running the postinst script reveals the real issue (missing user, full disk, etc).

Lock file contention during cloud-init

# apt hangs on boot because cloud-init is still running
lsof /var/lib/dpkg/lock-frontend
# COMMAND   PID USER   FD   TYPE DEVICE SIZE/OFF  NODE NAME
# apt-get  1234 root    5uW  REG  252,1        0 12345 /var/lib/dpkg/lock-frontend

# Wait for cloud-init to finish (correct fix)
cloud-init status --wait
# status: done

# Now safe to run apt
apt update && apt install -y jq

Gotcha: Never kill -9 the process holding the dpkg lock — it may leave the package database in a half-written state. Always wait for cloud-init to finish or use cloud-init status --wait. If you must break the lock, run dpkg --configure -a immediately after to repair the database.

Pin a package to prevent surprise upgrades

# Debian/Ubuntu — hold
apt-mark hold nginx
# nginx set on hold.
apt-mark showhold
# nginx

# RHEL/CentOS — versionlock
dnf install -y python3-dnf-plugin-versionlock
dnf versionlock add nginx-1.24.0-1.el9
dnf versionlock list
# nginx-0:1.24.0-1.el9.*

Remember: Package pinning mnemonic: H for Hold (Debian apt-mark hold), V for Versionlock (RHEL dnf versionlock). Both prevent the package from being upgraded during apt upgrade or dnf update. Forget to pin and your next automated patching window upgrades the one package you needed frozen.

Rollback a bad update (RHEL)

# See what happened
dnf history
# ID  | Command               | Date and time    | Actions | Altered
# 127 | upgrade --security    | 2024-03-14 02:00 | Upgrade |   12

dnf history info 127
# Upgraded: openssl-1:3.0.7-24.el9.x86_64  → openssl-1:3.0.7-25.el9.x86_64

# Roll it back
dnf history undo 127 -y

Compare package sets across two hosts

# Debian
diff <(ssh web-01 "dpkg-query -W -f='\${Package}\t\${Version}\n'" | sort) \
     <(ssh web-02 "dpkg-query -W -f='\${Package}\t\${Version}\n'" | sort)
# < libssl3    3.0.13-0ubuntu3
# > libssl3    3.0.13-0ubuntu3.1

# RHEL
diff <(ssh web-01 "rpm -qa --qf '%{NAME}\t%{VERSION}-%{RELEASE}\n'" | sort) \
     <(ssh web-02 "rpm -qa --qf '%{NAME}\t%{VERSION}-%{RELEASE}\n'" | sort)

Security-only updates

# Debian/Ubuntu
apt list --upgradable 2>/dev/null | grep -i security

# RHEL
dnf updateinfo list security
# RHSA-2024:1234  Important/Sec.  openssl-3.0.7-25.el9.x86_64
dnf upgrade --security -y

Clean up disk space eaten by package cache

# Debian — show before and after
du -sh /var/cache/apt/archives/
# 847M    /var/cache/apt/archives/
apt clean
du -sh /var/cache/apt/archives/
# 40K     /var/cache/apt/archives/

# Remove orphan dependencies
apt autoremove -y

# RHEL
du -sh /var/cache/dnf/
# 612M    /var/cache/dnf/
dnf clean all

Add a third-party repo safely (modern Ubuntu)

# Download GPG key to keyring (NOT apt-key)
curl -fsSL https://packages.example.com/gpg.key | \
    gpg --dearmor -o /usr/share/keyrings/example.gpg

# Add repo with signed-by reference
cat <<'EOF' > /etc/apt/sources.list.d/example.list
deb [signed-by=/usr/share/keyrings/example.gpg] https://packages.example.com/apt stable main
EOF

apt update
apt install -y example-tool

Under the hood: The signed-by field in the apt sources entry pins that repo to a specific GPG key, preventing a compromised key from one repo from signing packages in another. The old apt-key add approach (deprecated in Debian 12/Ubuntu 22.04) added keys to a global keyring trusted by all repos — a single compromised key could inject packages into any repo.

Idempotent installs in shell automation

ensure_installed() {
    local pkg="$1"
    if ! dpkg -s "$pkg" &>/dev/null; then
        apt-get install -y "$pkg"
    fi
}

ensure_installed curl
ensure_installed jq
ensure_installed vim