Remediation: Backup Job Failing, iSCSI Target Unreachable, Fix Is VLAN Config¶

Immediate Fix (Networking — Domain C)¶

The fix is to add the iSCSI port to the inter-VLAN ACL.

Step 1: Add iSCSI to the ACL¶

switch# configure terminal
switch(config)# ip access-list extended VLAN10-to-VLAN20
switch(config-acl)# 25 permit tcp 10.0.10.0 0.0.0.255 10.0.20.0 0.0.0.255 eq 3260
switch(config-acl)# 26 remark iSCSI - added 2026-03-19 fix for DC-hardening-phase2 oversight
switch(config-acl)# end
switch# write memory

Step 2: Verify iSCSI connectivity¶

# From db-primary-01
$ nc -zv iscsi-san.storage.internal 3260 -w 5
Connection to iscsi-san.storage.internal (10.0.20.50) 3260 port [tcp/*] succeeded!

Step 3: Re-establish iSCSI session and mount¶

$ iscsiadm -m discovery -t sendtargets -p iscsi-san.storage.internal:3260
10.0.20.50:3260,1 iqn.2023-01.com.storage:backup-lun-04

$ iscsiadm -m node --login
Logging in to [iface: default, target: iqn.2023-01.com.storage:backup-lun-04, portal: 10.0.20.50,3260]
Login to ... successful.

$ mount /mnt/backup
$ df -h /mnt/backup
Filesystem      Size  Used Avail Use% Mounted on
/dev/sdb1       500G  142G  333G  30% /mnt/backup

Step 4: Run the missed backup manually¶

$ pg_dump -h localhost -U postgres -Z 5 -Fc production > /mnt/backup/pg_backup_20260319.sql.gz
$ ls -lh /mnt/backup/pg_backup_20260319.sql.gz
-rw-r--r-- 1 postgres postgres 4.2G Mar 19 03:15 pg_backup_20260319.sql.gz

Step 5: Repeat for other affected servers¶

$ for host in db-primary-02 db-replica-01; do
    ssh $host "iscsiadm -m discovery -t sendtargets -p iscsi-san.storage.internal:3260 && \
               iscsiadm -m node --login && \
               mount /mnt/backup && \
               echo '$host: OK'"
done
db-primary-02: OK
db-replica-01: OK

Verification¶

Domain A (Linux Ops) — Backups running, mounts stable¶

$ mount | grep backup
/dev/sdb1 on /mnt/backup type ext4 (rw,relatime,_netdev)

$ iscsiadm -m session
tcp: [1] 10.0.20.50:3260,1 iqn.2023-01.com.storage:backup-lun-04 (non-flash)

Domain B (Datacenter Ops) — Storage array showing active sessions¶

$ ssh admin@iscsi-san.storage.internal
storage> show iscsi sessions
Active Sessions: 3
  1. db-primary-01 -> backup-lun-04
  2. db-primary-02 -> backup-lun-05
  3. db-replica-01 -> backup-lun-06

Domain C (Networking) — ACL includes iSCSI¶

switch# show access-lists VLAN10-to-VLAN20 | include 3260
    25 permit tcp 10.0.10.0 0.0.0.255 10.0.20.0 0.0.0.255 eq 3260 (47 matches)

Prevention¶

Monitoring: Add iSCSI session health monitoring. Alert when any server loses its iSCSI session or when a backup mount is not available.

- alert: ISCSISessionDown
  expr: node_iscsi_sessions == 0
  for: 10m
  labels:
    severity: critical
  annotations:
    summary: "No active iSCSI sessions on {{ $labels.instance }}"

Runbook: Every datacenter hardening ACL change must include a traffic audit — check existing sessions and connections before replacing permissive rules. Use show ip access-list hit counters to identify active traffic before restricting.
Architecture: Move iSCSI traffic to a dedicated storage VLAN and physical network (storage fabric) that is separate from the general server VLAN. This removes the dependency on inter-VLAN ACLs for storage traffic.