Skip to content

Anti-Primer: Falco

Everything that can go wrong, will — and in this story, it does.

The Setup

A security engineer is implementing Falco controls for a production environment ahead of a compliance audit. The deadline is next week, and the engineer is working through a checklist of security requirements, rushing to clear each item.

The Timeline

Hour 0: Overly Permissive Initial Config

Starts with an allow-all policy intending to restrict later, but 'later' never comes. The deadline was looming, and this seemed like the fastest path forward. But the result is audit discovers wide-open permissions; the 'temporary' config has been running for 3 months.

Footgun #1: Overly Permissive Initial Config — starts with an allow-all policy intending to restrict later, but 'later' never comes, leading to audit discovers wide-open permissions; the 'temporary' config has been running for 3 months.

Nobody notices yet. The engineer moves on to the next task.

Hour 1: Secrets in Plain Text

Stores API keys and passwords in a config file committed to version control. Under time pressure, the team chose speed over caution. But the result is secrets are visible in git history forever; credential rotation required across all environments.

Footgun #2: Secrets in Plain Text — stores API keys and passwords in a config file committed to version control, leading to secrets are visible in git history forever; credential rotation required across all environments.

The first mistake is still invisible, making the next shortcut feel justified.

Hour 2: No Monitoring for Security Events

Implements controls but does not set up alerting for violations. Nobody pushed back because the shortcut looked harmless in the moment. But the result is a breach occurs and goes undetected for weeks because nobody is watching the audit logs.

Footgun #3: No Monitoring for Security Events — implements controls but does not set up alerting for violations, leading to a breach occurs and goes undetected for weeks because nobody is watching the audit logs.

Pressure is mounting. The team is behind schedule and cutting more corners.

Hour 3: Expired Certificates or Keys

Deploys certificates without setting up renewal automation or expiry monitoring. The team had gotten away with similar shortcuts before, so nobody raised a flag. But the result is certificate expires at 2 AM on a Saturday; the entire service goes down until someone renews manually.

Footgun #4: Expired Certificates or Keys — deploys certificates without setting up renewal automation or expiry monitoring, leading to certificate expires at 2 AM on a Saturday; the entire service goes down until someone renews manually.

By hour 3, the compounding failures have reached critical mass. Pages fire. The war room fills up. The team scrambles to understand what went wrong while the system burns.

The Postmortem

Root Cause Chain

# Mistake Consequence Could Have Been Prevented By
1 Overly Permissive Initial Config Audit discovers wide-open permissions; the 'temporary' config has been running for 3 months Primer: Start with deny-all and explicitly allow what is needed; never invert this
2 Secrets in Plain Text Secrets are visible in git history forever; credential rotation required across all environments Primer: Use a secrets manager; never commit secrets to repos; use pre-commit hooks to scan
3 No Monitoring for Security Events A breach occurs and goes undetected for weeks because nobody is watching the audit logs Primer: Alerts on security events: failed auth, privilege escalation, policy violations
4 Expired Certificates or Keys Certificate expires at 2 AM on a Saturday; the entire service goes down until someone renews manually Primer: Automated certificate renewal (cert-manager, Let's Encrypt); monitor expiry dates

Damage Report

  • Downtime: Potentially none immediately, but exposure window may span days or weeks
  • Data loss: Credentials or sensitive data may have been exposed
  • Customer impact: If a breach occurred, full customer notification may be required
  • Engineering time to remediate: 20-40 engineer-hours for audit, rotation, and hardening
  • Reputation cost: Severe: compliance violations, possible regulatory notification, customer trust damage

What the Primer Teaches

  • Footgun #1: If the engineer had read the primer, section on overly permissive initial config, they would have learned: Start with deny-all and explicitly allow what is needed; never invert this.
  • Footgun #2: If the engineer had read the primer, section on secrets in plain text, they would have learned: Use a secrets manager; never commit secrets to repos; use pre-commit hooks to scan.
  • Footgun #3: If the engineer had read the primer, section on no monitoring for security events, they would have learned: Alerts on security events: failed auth, privilege escalation, policy violations.
  • Footgun #4: If the engineer had read the primer, section on expired certificates or keys, they would have learned: Automated certificate renewal (cert-manager, Let's Encrypt); monitor expiry dates.

Cross-References