Portal | Level: L2: Operations | Topics: Runtime Security with Falco | Domain: Security

Runtime Security with Falco — Primer

Why This Matters

Image scanning tells you what vulnerabilities exist in your software supply chain. It cannot tell you what your containers are doing at runtime. A scanner passes an image with no CVEs — and then that container, once running, opens a reverse shell, reads /etc/shadow, makes unexpected outbound connections, or escalates privileges via a zero-day.

Falco solves the runtime blindspot. It hooks into the kernel (via eBPF or a kernel module) and watches every syscall made by every process on every node. It evaluates those syscalls against a policy expressed as rules. When a rule matches — container spawns a shell, sensitive file is read, binary is executed from /tmp — Falco fires an alert.

This is the difference between supply chain security (what's in the box) and runtime security (what the box is doing).

Who made it: Falco was created by Sysdig in 2016 and donated to the CNCF in 2018, where it became the first runtime security project. It graduated to CNCF Incubation status in 2020. The name "Falco" comes from the Latin word for falcon — a bird known for its sharp eyesight and ability to detect prey from great distances.

---

Core Concepts

1. Architecture

Kernel syscalls
  └─► Falco driver (eBPF probe OR kernel module)
        └─► libscap (syscall capture)
              └─► libsinsp (event enrichment: container metadata, k8s metadata)
                    └─► Falco engine (rules evaluation)
                          └─► Outputs: stdout / file / gRPC / program
                                └─► Falcosidekick (fan-out to Slack, PagerDuty, Loki, etc.)

Two kernel instrumentation modes:

Mode	Mechanism	Pros	Cons
Kernel module (.ko)	Traditional kernel module	Lowest overhead	Requires kernel headers, may break on kernel upgrade
eBPF probe (modern)	BPF CO-RE (Compile Once, Run Everywhere)	No kernel headers needed, safer	Requires kernel 5.8+ for modern probe
eBPF legacy probe	BPF without CO-RE	Works on older kernels (4.14+)	Larger compiled artifact

In Kubernetes, the eBPF modern probe is the recommended default for kernels ≥ 5.8.

Under the hood: Falco intercepts syscalls at the kernel level, not at the container runtime level. This means it sees everything — even if an attacker escapes the container, Falco still observes the syscalls on the host. The eBPF probe uses tracepoint/raw_syscalls/sys_enter and sys_exit to capture every syscall with its arguments, enriched with container and Kubernetes metadata by libsinsp.

2. Installation

Helm (Kubernetes, DaemonSet):

helm repo add falcosecurity https://falcosecurity.github.io/charts
helm repo update

# Install with eBPF modern probe (recommended)
helm install falco falcosecurity/falco \
  --namespace falco \
  --create-namespace \
  --set driver.kind=modern_ebpf \
  --set falcosidekick.enabled=true \
  --set falcosidekick.webui.enabled=true

# Verify
kubectl get pods -n falco
# NAME                                  READY   STATUS    RESTARTS   AGE
# falco-xxxxx (one per node, DaemonSet)   2/2     Running   0          2m
# falcosidekick-xxxxx                     1/1     Running   0          2m
# falcosidekick-ui-xxxxx                  1/1     Running   0          2m

# Tail logs (watch for alerts)
kubectl logs -n falco -l app.kubernetes.io/name=falco -f

Helm values for production:

# falco-values.yaml
driver:
  kind: modern_ebpf

falco:
  grpc:
    enabled: true
    bind_address: "0.0.0.0:5060"
    threadiness: 8
  grpc_output:
    enabled: true
  json_output: true
  log_level: info
  buffered_outputs: false
  syscall_event_drops:
    actions:
      - log
      - alert
    rate: 0.03333
    max_burst: 1

  # Custom rules file
  rules_file:
    - /etc/falco/falco_rules.yaml
    - /etc/falco/falco_rules.local.yaml
    - /etc/falco/k8s_audit_rules.yaml
    - /etc/falco/rules.d

falcosidekick:
  enabled: true
  config:
    slack:
      webhookurl: "https://hooks.slack.com/services/T.../B.../..."
      channel: "#security-alerts"
      minimumpriority: "warning"
    pagerduty:
      routingKey: "your-pagerduty-routing-key"
      minimumpriority: "critical"
    loki:
      hostport: "http://loki.monitoring.svc:3100"
      minimumpriority: "debug"

  webui:
    enabled: true
    replicaCount: 1

3. Falco Rules Language

A Falco rule has five key fields:

- rule: Shell Spawned in Container
  desc: A shell was spawned inside a container. This may indicate an attacker
        gaining interactive access or a debugging session left open in production.
  condition: >
    spawned_process
    and container
    and shell_procs
    and not container.image.repository in (allowed_shell_containers)
  output: >
    Shell spawned in container
    (evt.time=%evt.time
     user=%user.name
     container=%container.name
     image=%container.image.repository:%container.image.tag
     shell=%proc.name
     parent=%proc.pname
     cmdline=%proc.cmdline
     pid=%proc.pid
     k8s.ns=%k8s.ns.name
     k8s.pod=%k8s.pod.name)
  priority: WARNING
  tags: [container, shell, mitre_execution]

Field	Purpose
rule	Unique rule name
desc	Human description
condition	Boolean expression over syscall fields
output	Alert message template with field substitutions
priority	EMERGENCY, ALERT, CRITICAL, ERROR, WARNING, NOTICE, INFO, DEBUG
tags	Categorization for filtering and routing

4. Macros and Lists

Macros and lists let you factor out reusable logic:

# Lists — sets of values for use in conditions
- list: shell_binaries
  items: [bash, csh, ksh, sh, tcsh, zsh, dash]

- list: sensitive_files
  items: [/etc/shadow, /etc/sudoers, /root/.ssh/authorized_keys, /etc/kubernetes/admin.conf]

- list: allowed_shell_containers
  items: [debug-tools, kubectl-debug, my-init-container]

# Macros — reusable condition fragments
- macro: spawned_process
  condition: evt.type = execve and evt.dir = <

- macro: shell_procs
  condition: proc.name in (shell_binaries)

- macro: container
  condition: (container.id != host)

- macro: sensitive_file_read
  condition: >
    open_read
    and fd.name in (sensitive_files)
    and not proc.name in (trusted_procs)

5. Built-In Rule Sets

Falco ships with rules covering the most common attack patterns:

Shell in container:

- rule: Terminal shell in container
  condition: >
    spawned_process
    and container
    and shell_procs
    and proc.tty != 0
    and container_entrypoint
  output: >
    A shell was spawned in a container with an attached terminal
    (user=%user.name %container.info shell=%proc.name parent=%proc.pname
     cmdline=%proc.cmdline terminal=%proc.tty container_id=%container.id image=%container.image.repository)
  priority: NOTICE
  tags: [container, shell, mitre_execution]

Sensitive file read:

- rule: Read sensitive file untrusted
  condition: >
    open_read
    and sensitive_files
    and not trusted_programs_reading_sensitive_files
    and not user_known_read_sensitive_files_activities
  output: >
    Sensitive file opened for reading by non-trusted program
    (user=%user.name user_loginuid=%user.loginuid program=%proc.name
     command=%proc.cmdline file=%fd.name parent=%proc.pname
     gparent=%proc.aname[2] container_id=%container.id image=%container.image.repository)
  priority: WARNING
  tags: [filesystem, mitre_credential_access]

Privilege escalation — setuid/setgid:

- rule: Privilege escalation using nsenter
  condition: >
    spawned_process
    and proc.name = nsenter
    and not proc.pname in (allowed_nsenter_callers)
  output: >
    Possible privilege escalation via nsenter
    (user=%user.name user_loginuid=%user.loginuid
     proc.name=%proc.name proc.args=%proc.args
     container_id=%container.id image=%container.image.repository)
  priority: WARNING
  tags: [process, privilege_escalation, mitre_privilege_escalation]

Unexpected network connection:

- rule: Unexpected outbound connection destination
  condition: >
    outbound
    and not ec2_metadata_servers
    and not allowed_outbound_destination_domains
    and container
    and not proc.name in (known_network_tools)
  output: >
    Unexpected outbound connection
    (user=%user.name command=%proc.cmdline
     connection=%fd.name container_id=%container.id
     image=%container.image.repository)
  priority: NOTICE
  tags: [network, mitre_exfiltration]

6. Writing Custom Rules

Pattern: detect binary execution from writable directory:

- rule: Run shell from /tmp
  desc: A process executed a binary from /tmp — common malware deployment pattern
  condition: >
    spawned_process
    and (proc.exe startswith /tmp/ or proc.exe startswith /dev/shm/)
    and container
  output: >
    Binary executed from writable dir
    (user=%user.name command=%proc.cmdline
     exe=%proc.exe parent=%proc.pname
     container=%container.name image=%container.image.repository
     k8s.pod=%k8s.pod.name k8s.ns=%k8s.ns.name)
  priority: ERROR
  tags: [process, malware, mitre_execution]

Pattern: detect crypto mining indicators:

- list: crypto_mining_domains
  items: [pool.minexmr.com, xmrpool.eu, supportxmr.com, "*.nicehash.com", "*.miningpoolhub.com"]

- rule: Crypto mining outbound connection
  desc: Process connecting to known crypto mining pool
  condition: >
    outbound
    and fd.sip.name in (crypto_mining_domains)
  output: >
    Crypto mining connection detected
    (user=%user.name command=%proc.cmdline destination=%fd.name
     container=%container.name image=%container.image.repository)
  priority: CRITICAL
  tags: [network, crypto_mining, mitre_impact]

Pattern: detect kubectl exec into production pods:

- rule: Exec into production pod
  desc: kubectl exec was used to open a shell into a production pod
  condition: >
    ka.verb = create
    and ka.target.subresource = exec
    and ka.target.namespace = production
    and not ka.user.name in (allowed_exec_users)
  output: >
    kubectl exec into production pod
    (user=%ka.user.name pod=%ka.target.name
     namespace=%ka.target.namespace
     container=%ka.target.container.name)
  priority: WARNING
  tags: [k8s_audit, mitre_execution]
  source: k8s_audit

7. False Positive Tuning

The default Falco rules fire frequently in normal workloads. Systematic tuning approach:

Step 1 — identify noisy rules:

# Run in permissive mode, count alerts per rule over 24h
kubectl logs -n falco -l app.kubernetes.io/name=falco --since=24h | \
  jq -r '.rule' | sort | uniq -c | sort -rn | head -20

Gotcha: Never edit falco_rules.yaml directly — it is overwritten on every Falco upgrade. Always put customizations in falco_rules.local.yaml, which is loaded after the base rules and takes precedence. Use append: true on lists and macros to add to existing definitions rather than replacing them.

Step 2 — override rules in falco_rules.local.yaml (never edit the base rules file):

# falco_rules.local.yaml — local overrides, loaded after base rules

# Expand an existing list to add allowed tools
- list: allowed_shell_containers
  items: [my-debug-tools, vault-agent-init, linkerd-proxy-init]
  append: true   # adds to the existing list rather than replacing it

# Override a macro to exclude our trusted processes
- macro: trusted_programs_reading_sensitive_files
  condition: >
    (proc.name in (known_programs_reading_sensitive_files)
     or proc.name in (vault, consul-template, secret-init))
  append: true

# Disable a rule entirely for a specific workload
- rule: Redirect STDOUT/STDIN to Network Connection in Container
  condition: never   # override condition to never-true effectively disables it

Step 3 — use exception blocks (Falco 0.28+):

- rule: Write below root
  exceptions:
    - name: known_root_files
      fields: [proc.name, fd.name]
      values:
        - [nginx, /var/run/nginx.pid]
        - [dockerd, /var/lib/docker]

8. Falco Sidekick

Falcosidekick receives Falco events via webhook/gRPC and fans them out to multiple outputs:

# falcosidekick-config.yaml
config:
  slack:
    webhookurl: "https://hooks.slack.com/..."
    channel: "#alerts"
    minimumpriority: warning
    messageformat: "long"

  pagerduty:
    routingKey: "xxxxx"
    minimumpriority: critical

  loki:
    hostport: "http://loki:3100"
    user: ""
    apikey: ""
    minimumpriority: debug
    customHeaders:
      X-Scope-OrgID: falco

  elasticsearch:
    hostport: "http://elasticsearch:9200"
    index: falco
    type: event
    minimumpriority: debug

  opsgenie:
    apikey: "xxxxx"
    region: us
    minimumpriority: critical

  webhook:
    address: "http://my-soar-platform/api/falco"
    method: POST
    minimumpriority: error
    customHeaders:
      Authorization: "Bearer my-token"

9. Threat Detection Patterns

MITRE Tactic	Falco Detection	Rule Example
> Remember: Falco maps to the MITRE ATT&CK framework. The tags field on each rule includes MITRE tactic references (e.g., mitre_execution, mitre_privilege_escalation). This lets you filter alerts by attack stage and build detection coverage maps against the ATT&CK matrix for containers.

| Initial Access | Container escape | Detect host filesystem write from container | | Execution | Shell in container | Terminal shell in container (built-in) | | Persistence | Cron modification | Write to /etc/cron* from container | | Privilege Escalation | setuid binary exec | Set Setuid or Setgid bit (built-in) | | Defense Evasion | Log deletion | Write to /var/log from unexpected process | | Credential Access | Secret file read | Read sensitive file untrusted (built-in) | | Discovery | Container enumeration | Unexpected kubectl get, docker ps | | Lateral Movement | SSH from container | Outbound connection on port 22 | | Exfiltration | Data to cloud storage | Outbound to S3/GCS from unexpected container | | Impact | Crypto mining | Connection to known mining pools |

---

Quick Reference

# Install Falco on Kubernetes
helm install falco falcosecurity/falco \
  --namespace falco --create-namespace \
  --set driver.kind=modern_ebpf \
  --set falcosidekick.enabled=true

# Watch alerts live
kubectl logs -n falco -l app.kubernetes.io/name=falco -f | jq .

# Count alerts by rule (last 24h)
kubectl logs -n falco -l app.kubernetes.io/name=falco --since=24h | \
  jq -r '.rule' | sort | uniq -c | sort -rn

# Validate a custom rules file
falco --validate /etc/falco/falco_rules.local.yaml

# List all loaded rules
falco --list

# Test a rule fires (spawn a shell in a test container)
kubectl run test --image=alpine --rm -it -- sh

# Check Falcosidekick health
kubectl port-forward -n falco svc/falco-falcosidekick 2801:2801
curl http://localhost:2801/ping

---

Wiki Navigation

Prerequisites

• Kubernetes Ops (Production) (Topic Pack, L2)
• eBPF & Modern Linux Observability (Topic Pack, L3)

Pages that link here

• Anti-Primer: Falco
• Certification Prep: CKS — Certified Kubernetes Security Specialist
• Runtime Security with Falco
• eBPF & Modern Linux Observability