Remediation: API Latency Spike, BGP Route Leak, Fix Is Network ACL¶

Immediate Fix (Security — Domain C)¶

The fix is to replace the BGP prefix-list filter with a proper inbound ACL that achieves the security intent without breaking routing.

Step 1: Remove the BGP prefix-list filter¶

router# configure terminal
router(config)# no ip prefix-list DENY-PARTNER-ROUTES seq 10 deny 198.51.100.0/24
router(config)# end
router# clear ip bgp 203.0.114.1 soft in

Step 2: Verify the preferred route is restored¶

router# show ip bgp 198.51.100.0/24
BGP routing table entry for 198.51.100.0/24
  Paths: (2 available, best #2)
    Path 1: AS_PATH 64501 64999 64700
      Next hop: 203.0.113.1 (ISP-A via transit)
      MED: 200, LOCAL_PREF: 100
    Path 2: AS_PATH 64502 64700
      Next hop: 203.0.114.1 (ISP-B direct peering)
      MED: 50, LOCAL_PREF: 150
      Origin: IGP, valid, best   **<-- Now active**

Step 3: Implement the correct security control — inbound ACL¶

router# configure terminal
router(config)# ip access-list extended BLOCK-PARTNER-INBOUND-SCANS
router(config-acl)# remark Block inbound scanning from partner range per SEC-4471
router(config-acl)# deny tcp 198.51.100.0 0.0.0.255 any range 1 1023 log
router(config-acl)# deny udp 198.51.100.0 0.0.0.255 any range 1 1023 log
router(config-acl)# permit ip any any
router(config-acl)# exit
router(config)# interface GigabitEthernet0/1
router(config-if)# ip access-group BLOCK-PARTNER-INBOUND-SCANS in
router(config-if)# end
router# write memory

This blocks inbound scanning from the partner's range (the actual security intent) without affecting outbound BGP route selection.

Verification¶

Domain A (Observability) — Latency back to normal¶

# Prometheus query after fix:
# histogram_quantile(0.99, rate(http_request_duration_seconds_bucket{service="checkout-api",handler="/v1/checkout"}[5m]))
# Result: 0.172 seconds — back to baseline

# Error rate:
# rate(http_requests_total{service="checkout-api",status=~"5.."}[5m]) / rate(http_requests_total{service="checkout-api"}[5m])
# Result: 0.001 — back to baseline 0.1%

Domain B (Networking) — Route path correct¶

$ kubectl exec -it checkout-api-6b5d8c9f-x2k4j -n prod -- traceroute -n 198.51.100.42
traceroute to 198.51.100.42, 30 hops max
 1  10.0.0.1      0.5 ms
 2  10.0.1.1      0.8 ms
 3  172.16.0.1    1.2 ms
 4  203.0.114.1   2.1 ms    # ISP-B (direct peering)
 5  198.51.100.1  14.2 ms   # Partner's edge
 6  198.51.100.42 16.8 ms   # Partner's server

6 hops, 17ms — back to the direct peering path.

Domain C (Security) — Inbound scanning blocked¶

router# show access-lists BLOCK-PARTNER-INBOUND-SCANS
Extended IP access list BLOCK-PARTNER-INBOUND-SCANS
    10 deny tcp 198.51.100.0 0.0.0.255 any range 1 1023 log (23 matches)
    20 deny udp 198.51.100.0 0.0.0.255 any range 1 1023 log (7 matches)
    30 permit ip any any (1284721 matches)

The ACL is active and blocking inbound scans while permitting all other traffic.

Prevention¶

Monitoring: Add a BGP route change alert that fires when the active path to critical partner prefixes changes. Monitor AS path length as a proxy for route quality.

- alert: BGPPathChanged
  expr: bgp_path_as_count{prefix="198.51.100.0/24"} > 3
  for: 5m
  labels:
    severity: warning
  annotations:
    summary: "BGP path to partner network has changed — AS path length increased"

Runbook: Security tickets that involve IP range blocking must specify the control type (inbound ACL, outbound ACL, or BGP filter) and must be reviewed by the network team for routing impact before implementation.
Architecture: Separate the security ACL from BGP route policy. Use a firewall or a separate ACL on the interface for traffic filtering. BGP prefix-lists should only be used for route policy, never for security filtering.