Skip to content

Package Management - Primer

Why This Matters

Package management is day-1 Linux operations. Every server you touch runs packages you did not compile, and every production incident eventually traces back to a version mismatch, a broken dependency, or a repository you forgot to pin. Getting this wrong means silent drift across your fleet, security patches that never land, or a 3 AM apt upgrade that removes half your stack.

Core Concepts

1. The Two Families: Debian vs Red Hat

Linux package management splits along two lineages. You will encounter both in any fleet of meaningful size.

Aspect Debian/Ubuntu RHEL/CentOS/Fedora
Low-level tool dpkg rpm
High-level tool apt / apt-get yum / dnf
Package format .deb .rpm
Config directory /etc/apt/ /etc/yum.repos.d/
Package database /var/lib/dpkg/ /var/lib/rpm/
Cache directory /var/cache/apt/archives/ /var/cache/yum/ or /var/cache/dnf/

Name origin: apt stands for Advanced Package Tool. dpkg stands for Debian Package. rpm stands for RPM Package Manager (a recursive acronym, originally Red Hat Package Manager). dnf stands for Dandified YUM, and yum stands for Yellowdog Updater, Modified — originally written for Yellow Dog Linux, a PowerPC distribution.

dnf replaced yum starting with Fedora 22 and RHEL 8. On modern RHEL/CentOS Stream, yum is a symlink to dnf. Use dnf in new automation; support yum for legacy hosts.

2. Low-Level Tools: dpkg and rpm

These operate on individual package files. They do not resolve dependencies. Use them for inspection, forensics, and emergencies — not routine installs.

# Install a local .deb (no dependency resolution)
dpkg -i /tmp/custom-agent_2.4.1_amd64.deb

# If dependencies are broken after dpkg -i:
apt-get install -f

# List all installed packages (Debian)
dpkg -l | grep nginx

# Which package owns a file? (Debian)
dpkg -S /usr/sbin/nginx
# nginx-core: /usr/sbin/nginx

# Install a local .rpm
rpm -ivh /tmp/custom-agent-2.4.1.x86_64.rpm

# Query an installed package (Red Hat)
rpm -qi nginx

# Which package owns a file? (Red Hat)
rpm -qf /usr/sbin/nginx

# List all files in a package (Red Hat)
rpm -ql nginx

Debug clue: When dpkg -i leaves a package in a broken state, you will see it marked as iF (installed, failed config) or iU (installed, unpacked but not configured) in dpkg -l output. The second and third characters in the status code tell you: i=installed, c=config-files only, U=unpacked, F=half-configured, H=half-installed.

Gotcha: dpkg -i will fail silently on missing dependencies and leave the package in a half-configured state. Always follow with apt-get install -f or better yet, use apt install ./package.deb which handles dependencies.

3. High-Level Tools: apt and dnf

These are what you use 99% of the time. They resolve dependencies, fetch from repositories, and handle upgrades.

# --- Debian/Ubuntu ---

# Update package index (does NOT upgrade anything)
apt update

# Upgrade all packages (safe — never removes packages)
apt upgrade -y

# Full upgrade (may remove packages to resolve conflicts)
apt full-upgrade -y

# Install a specific package
apt install -y nginx=1.24.0-1ubuntu1

# Remove package (keep config files)
apt remove nginx

# Remove package AND config files
apt purge nginx

# --- RHEL/CentOS/Fedora ---

# Refresh metadata
dnf check-update

# Install a package
dnf install -y nginx-1.24.0-1.el9

# Upgrade all packages
dnf upgrade -y

# Remove a package
dnf remove nginx

# Downgrade a package
dnf downgrade nginx-1.22.0-1.el9

Production rule: Never run apt upgrade or dnf upgrade interactively on a production server. Use automation (Ansible, unattended-upgrades) with a tested package set.

4. Searching and Listing

# Search for packages by name or description (Debian)
apt search "^nginx"
apt-cache search nginx

# Show package details before installing (Debian)
apt show nginx
apt-cache policy nginx
# Output:
# nginx:
#   Installed: 1.24.0-1ubuntu1
#   Candidate: 1.24.0-2ubuntu1
#   Version table:
#      1.24.0-2ubuntu1 500
#         500 http://archive.ubuntu.com/ubuntu noble/main amd64 Packages
#   *** 1.24.0-1ubuntu1 100
#         100 /var/lib/dpkg/status

# List installed packages matching a pattern (Debian)
apt list --installed 2>/dev/null | grep nginx

# Search for packages (Red Hat)
dnf search nginx

# Show package info (Red Hat)
dnf info nginx

# List installed packages (Red Hat)
dnf list installed | grep nginx

# List available versions of a package (Red Hat)
dnf list available nginx --showduplicates

5. Repositories

Repositories are where packages come from. Misconfigured repos are one of the top causes of fleet drift.

# --- Debian/Ubuntu ---
# Main repo config
cat /etc/apt/sources.list

# Additional repos (preferred location on modern Ubuntu)
ls /etc/apt/sources.list.d/

# Add a third-party repo
cat <<'EOF' > /etc/apt/sources.list.d/custom-app.list
deb [signed-by=/usr/share/keyrings/custom-app.gpg] https://packages.example.com/apt stable main
EOF
apt update

# --- RHEL/CentOS/Fedora ---
# List enabled repos
dnf repolist

# List all repos (including disabled)
dnf repolist all

# Add a repo
dnf config-manager --add-repo https://packages.example.com/rpm/stable/

# Enable/disable a repo
dnf config-manager --set-enabled epel
dnf config-manager --set-disabled epel-testing

Timeline: apt-key was deprecated in Debian 11 (2021) and Ubuntu 22.04 (2022). The new signed-by approach ties each repository to a specific GPG key, so a compromised third-party key cannot sign packages from other repositories. This is a significant security improvement over the old global keyring model.

Gotcha: On Ubuntu 22.04+, apt-key is deprecated. Use signed-by in the sources list pointing to a keyring file in /usr/share/keyrings/. Old tutorials still show apt-key add — this will break on modern systems.

6. GPG Keys and Repository Trust

Package managers verify signatures to ensure packages have not been tampered with. Skipping this is a security incident waiting to happen.

# --- Debian/Ubuntu (modern approach) ---

# Download and install a GPG key to the keyring directory
curl -fsSL https://packages.example.com/gpg.key | \
    gpg --dearmor -o /usr/share/keyrings/example.gpg

# Reference in sources list
echo "deb [signed-by=/usr/share/keyrings/example.gpg] https://packages.example.com/apt stable main" \
    > /etc/apt/sources.list.d/example.list

# --- RHEL/CentOS/Fedora ---

# Import a GPG key
rpm --import https://packages.example.com/RPM-GPG-KEY-example

# Verify a package signature
rpm -K /tmp/example-1.0.0.x86_64.rpm
# example-1.0.0.x86_64.rpm: digests signatures OK

# List imported GPG keys
rpm -qa gpg-pubkey*

Production consequence: If you disable GPG checks (--nogpgcheck or apt --allow-unauthenticated), a compromised mirror can push arbitrary binaries to your fleet. Treat every --nogpgcheck in your automation as a security finding.

7. Version Pinning and Holding Packages

In production, you pin versions to prevent unintended upgrades that break your application.

# --- Debian/Ubuntu: hold a package ---
apt-mark hold nginx
# nginx set on hold.

# Check held packages
apt-mark showhold
# nginx

# Release the hold
apt-mark unhold nginx

# --- Debian/Ubuntu: pin via preferences ---
cat <<'EOF' > /etc/apt/preferences.d/pin-nginx
Package: nginx
Pin: version 1.24.0-1ubuntu1
Pin-Priority: 1001
EOF

# --- RHEL/CentOS/Fedora: exclude from updates ---
# In /etc/dnf/dnf.conf or /etc/yum.conf:
# excludepkgs=nginx*

# Or per-command:
dnf upgrade -y --exclude=nginx*

# Using the versionlock plugin (dnf)
dnf install -y python3-dnf-plugin-versionlock
dnf versionlock add nginx-1.24.0-1.el9
dnf versionlock list
dnf versionlock delete nginx
Priority Effect (Debian)
1001+ Forces install even if downgrade
990 Default for target release
500 Default for non-target
100 Installed package priority
-1 Never install

Gotcha: A held package blocks apt upgrade from completing if other packages depend on a newer version. Your unattended-upgrades will log errors and silently skip security patches for the entire transaction. Monitor /var/log/unattended-upgrades/.

8. Security Updates

Applying security patches is non-negotiable. The question is how to do it without breaking production.

# --- Debian/Ubuntu ---

# List available security updates
apt list --upgradable 2>/dev/null | grep -i security

# Install only security updates (requires unattended-upgrades)
apt install -y unattended-upgrades
dpkg-reconfigure -plow unattended-upgrades

# Check unattended-upgrades config
cat /etc/apt/apt.conf.d/50unattended-upgrades | grep -A5 Allowed-Origins

# Manual security-only upgrade
apt upgrade -y -o Dir::Etc::SourceList=/etc/apt/sources.list \
    -o Dir::Etc::SourceParts=/dev/null \
    -t $(lsb_release -cs)-security

# --- RHEL/CentOS/Fedora ---

# List security updates
dnf updateinfo list security

# Install security updates only
dnf upgrade --security -y

# Check advisory details
dnf updateinfo info RHSA-2024:1234

Production pattern: Stage security updates through dev -> staging -> prod with a 24-48h bake time between stages. Automate the promotion. Never skip staging for "just a security patch."

9. Cache Management

Package caches grow unbounded and will fill disks, especially in CI/CD or container builds.

# --- Debian/Ubuntu ---

# Show cache size
du -sh /var/cache/apt/archives/
# 847M    /var/cache/apt/archives/

# Clean only packages that can no longer be downloaded
apt autoclean

# Remove ALL cached packages
apt clean

# Remove unused dependencies
apt autoremove -y

# --- RHEL/CentOS/Fedora ---

# Show cache size
du -sh /var/cache/dnf/
# 612M    /var/cache/dnf/

# Clean all cached data
dnf clean all

# Clean only metadata
dnf clean metadata

# Clean only packages
dnf clean packages

In Dockerfiles: Always run apt clean && rm -rf /var/lib/apt/lists/* (or dnf clean all) in the same RUN layer as your install to keep image size down:

RUN apt-get update && \
    apt-get install -y --no-install-recommends nginx=1.24.0-1ubuntu1 && \
    apt-get clean && \
    rm -rf /var/lib/apt/lists/*

10. Troubleshooting Broken Packages

Broken packages are inevitable. Here is the triage sequence.

# --- Debian/Ubuntu ---

# Check for broken packages
dpkg --audit

# Fix broken dependencies
apt --fix-broken install

# Reconfigure packages that failed post-install scripts
dpkg --configure -a

# Force remove a completely broken package
dpkg --remove --force-remove-reinstreq <package>

# Check dpkg log for what happened
tail -50 /var/log/dpkg.log

# --- RHEL/CentOS/Fedora ---

# Check for problems
dnf check

# Rebuild the RPM database (nuclear option)
rpm --rebuilddb

# Check transaction history
dnf history
dnf history info 42

# Undo a specific transaction
dnf history undo 42

The 3 AM sequence for a broken dpkg: 1. dpkg --audit — what is broken? 2. apt --fix-broken install — let apt try to resolve it 3. dpkg --configure -a — finish pending configurations 4. If all else fails: dpkg --remove --force-remove-reinstreq <package> then reinstall 5. Check /var/log/dpkg.log and /var/log/apt/history.log for the timeline

Gotcha: rpm --rebuilddb rebuilds the database from headers of installed packages. If the headers are corrupt, you lose package tracking. Take a backup of /var/lib/rpm/ first.

11. Transaction History and Rollback

Knowing what changed and being able to undo it is critical for production.

# --- dnf transaction history ---
dnf history
# ID  | Command               | Date and time    | Actions | Altered
# 127 | upgrade --security    | 2024-03-14 02:00 | Upgrade |   12
# 126 | install nginx         | 2024-03-13 14:22 | Install |    4

dnf history info 127
dnf history undo 127 -y

# --- Debian/Ubuntu ---
# No built-in transaction rollback. Use:
cat /var/log/apt/history.log
# Or parse /var/log/dpkg.log

# Downgrade a specific package
apt install nginx=1.22.0-1ubuntu1

Interview tip: When asked about configuration drift across a fleet, the strong answer connects package management to automation: "Pin versions in your Ansible/Puppet/Chef manifests, use apt-mark hold or dnf versionlock to prevent unattended upgrades from breaking pins, and audit installed packages across hosts with dpkg-query -W or rpm -qa piped through a diff."

Production lesson: dnf history undo is one of the strongest arguments for RHEL-family in production. On Debian, you must track changes externally (Ansible, etckeeper, or your own tooling) because there is no native rollback.

12. Practical Patterns

Pattern: Idempotent package install in automation 

# Ansible-style check before install (shell script)
ensure_installed() {
    local pkg="$1"
    if ! dpkg -s "$pkg" &>/dev/null; then
        apt-get install -y "$pkg"
    fi
}

ensure_installed curl
ensure_installed jq

Pattern: Lock file contention 

# apt lock files: /var/lib/dpkg/lock-frontend, /var/lib/apt/lists/lock
# If another process holds the lock:
lsof /var/lib/dpkg/lock-frontend
# Kill the stale process or wait — never rm the lock file directly

# Common in cloud-init race conditions:
# cloud-init runs apt on boot, your automation runs apt at the same time
# Solution: wait for cloud-init to finish
cloud-init status --wait

Pattern: Audit what is installed 

# Full installed package list with versions (Debian)
dpkg-query -W -f='${Package}\t${Version}\n' > /tmp/packages-$(hostname).txt

# Full installed package list with versions (Red Hat)
rpm -qa --queryformat '%{NAME}\t%{VERSION}-%{RELEASE}\n' > /tmp/packages-$(hostname).txt

# Compare two hosts
diff <(ssh host1 "dpkg-query -W -f='\${Package}\t\${Version}\n'" | sort) \
     <(ssh host2 "dpkg-query -W -f='\${Package}\t\${Version}\n'" | sort)

Wiki Navigation

Related Content