Skip to content

Symptoms: Backup Job Failing, iSCSI Target Unreachable, Fix Is VLAN Config

Domains: linux_ops | datacenter_ops | networking Level: L3 Estimated time: 45 min

Initial Alert

Backup monitoring system fires at 02:30 UTC:

CRITICAL: Nightly backup FAILED — db-primary-01
Job: pg-backup-nightly
Duration: 0s (expected: 15-20 min)
Error: "pg_dump: could not open output file /mnt/backup/pg_backup_20260319.sql.gz: No such file or directory"

Follow-up alert:

CRITICAL: Mount point /mnt/backup is not mounted on db-primary-01
WARNING: Backup SLA breach — 2 consecutive nightly failures

Observable Symptoms

  • The backup job fails because /mnt/backup is not mounted.
  • mount | grep backup on db-primary-01 returns nothing — the filesystem is not mounted.
  • /etc/fstab shows the backup mount as an iSCSI target: iscsi-san.storage.internal:/backup-lun-04 /mnt/backup ext4 _netdev,defaults 0 0.
  • iscsiadm -m session shows no active iSCSI sessions.
  • ping iscsi-san.storage.internal succeeds (ICMP echo works).
  • The backup worked fine 3 days ago. Nothing changed on the database server.
  • Two other database servers (db-primary-02, db-replica-01) are also failing to mount their iSCSI backup volumes.

The Misleading Signal

A missing mount point with a backup failure looks like a Linux filesystem issue — maybe the iSCSI initiator service crashed, fstab is corrupted, or the mount was accidentally unmounted. The fact that ping to the storage array works makes it look like the storage target is up but the iSCSI service on the Linux host is broken. Engineers start investigating iscsid, checking iSCSI target discovery, and reviewing systemd mount units.