YAML, JSON & Config Formats - Street Ops¶
What experienced operators know about wrangling config formats in production that documentation glosses over.
Validating YAML Before Deploy¶
Never kubectl apply without validating first. YAML errors in production are preventable.
Gotcha: YAML's
Norway problem: the valueNO(unquoted) is parsed as booleanfalsein YAML 1.1. Country codes, yes/no fields, and on/off toggles silently become booleans. Always quote strings that could be misinterpreted:"NO","yes","on","off","true","null". YAML 1.2 (used by newer parsers) fixes this, but many tools still use 1.1.
yamllint as the First Gate¶
# Quick check with relaxed rules (catches indentation and syntax)
yamllint -d relaxed deployment.yaml
# Strict check with custom config
yamllint -d '{extends: default, rules: {line-length: {max: 200}, truthy: {check-keys: false}}}' deployment.yaml
# Check all manifests in a directory
find k8s/ -name '*.yaml' -exec yamllint -d relaxed {} +
# CI-friendly: non-zero exit on any warning
yamllint --strict deployment.yaml
Python One-Liner Validation¶
When yamllint isn't installed (bare containers, minimal CI images):
# Validate YAML syntax
python3 -c "import yaml; yaml.safe_load(open('deployment.yaml'))"
# Validate and print parsed output (catches subtle issues)
python3 -c "
import yaml, sys, json
with open(sys.argv[1]) as f:
docs = list(yaml.safe_load_all(f))
print(f'{len(docs)} document(s), valid YAML')
" deployment.yaml
# Validate all YAML files in a tree
find . -name '*.yaml' -o -name '*.yml' | while read f; do
python3 -c "import yaml; yaml.safe_load(open('$f'))" 2>&1 || echo "FAIL: $f"
done
Kubernetes-Specific Validation¶
# Dry-run against the API server (catches schema errors)
kubectl apply --dry-run=server -f deployment.yaml
# Client-side dry-run (no cluster needed, catches basic structural errors)
kubectl apply --dry-run=client -f deployment.yaml
# Validate with kubeconform (offline schema validation)
kubeconform -strict -summary deployment.yaml
# kubeval (older, still widely used)
kubeval --strict deployment.yaml
Converting Between Formats¶
YAML to JSON¶
# With yq
yq -o json deployment.yaml
# With Python (works everywhere)
python3 -c "import yaml, json, sys; print(json.dumps(yaml.safe_load(open(sys.argv[1])), indent=2))" config.yaml
# Multi-document YAML to JSON array
python3 -c "
import yaml, json, sys
docs = list(yaml.safe_load_all(open(sys.argv[1])))
print(json.dumps(docs, indent=2))
" multi-doc.yaml
JSON to YAML¶
# With yq
yq -P config.json # -P for pretty YAML
# With Python
python3 -c "import yaml, json, sys; print(yaml.dump(json.load(open(sys.argv[1])), default_flow_style=False))" config.json
TOML to JSON/YAML¶
# Python (toml or tomllib in 3.11+)
python3 -c "
import tomllib, json, sys
with open(sys.argv[1], 'rb') as f:
print(json.dumps(tomllib.load(f), indent=2, default=str))
" config.toml
# TOML to YAML (chain through JSON)
python3 -c "
import tomllib, yaml, sys
with open(sys.argv[1], 'rb') as f:
print(yaml.dump(tomllib.load(f), default_flow_style=False))
" config.toml
Finding Syntax Errors in Large YAML Files¶
When a 2000-line Helm values file has a syntax error and the parser just says "syntax error on line 847":
# Step 1: Get the line number from the parser
yamllint values.yaml 2>&1 | head -20
# Step 2: Look at context around the error
sed -n '840,855p' values.yaml
# Step 3: Common culprits to check
# - Tab characters (YAML forbids tabs for indentation)
grep -P '\t' values.yaml | head -10
# - Trailing whitespace on lines that should be empty
grep -n ' $' values.yaml
# - Mismatched indentation (find lines with odd indentation in a 2-space file)
awk '/^( {1}[^ ]| {3}[^ ]| {5}[^ ]| {7}[^ ])/' values.yaml
# Step 4: Binary search — split file and validate halves
head -500 values.yaml | python3 -c "import yaml, sys; yaml.safe_load(sys.stdin)"
tail -n +501 values.yaml | python3 -c "import yaml, sys; yaml.safe_load(sys.stdin)"
# Step 5: Check for invisible characters (copy-paste from docs/Slack/web)
cat -A values.yaml | sed -n '845,850p'
# Shows tabs as ^I, trailing spaces as visible, non-ASCII as M-
Debug clue: If a YAML file validates fine locally but fails in CI, check for invisible Unicode characters. Copy-pasting from Slack, Confluence, or web browsers often introduces non-breaking spaces (U+00A0) or zero-width characters that look identical to normal spaces but cause YAML parsers to choke.
Templating with envsubst¶
# Template file (config.template.yaml)
# apiVersion: v1
# kind: ConfigMap
# metadata:
# name: ${APP_NAME}-config
# namespace: ${NAMESPACE}
# data:
# database_url: "postgres://${DB_USER}:${DB_PASS}@${DB_HOST}:5432/${DB_NAME}"
# log_level: "${LOG_LEVEL:-info}" # default won't work — envsubst doesn't understand :-
# Generate from template
export APP_NAME=myapp NAMESPACE=production DB_USER=app DB_PASS=secret DB_HOST=db.internal DB_NAME=myapp LOG_LEVEL=warn
envsubst < config.template.yaml > config.yaml
# IMPORTANT: Restrict which variables are substituted
# Without restriction, ${LOG_LEVEL:-info} becomes empty (envsubst replaces the whole thing)
envsubst '$APP_NAME $NAMESPACE $DB_USER $DB_PASS $DB_HOST $DB_NAME $LOG_LEVEL' < config.template.yaml > config.yaml
# Pipe into kubectl
envsubst '$IMAGE_TAG $REPLICAS' < deployment.template.yaml | kubectl apply -f -
# Validate after substitution
envsubst < template.yaml | yamllint -d relaxed -
envsubst < template.yaml | kubectl apply --dry-run=client -f -
Merging YAML Files¶
yq Merge¶
# Override values (second file wins)
yq eval-all 'select(fileIndex == 0) * select(fileIndex == 1)' base.yaml override.yaml
# Deep merge (recursive, second file wins on conflicts)
yq eval-all 'select(fileIndex == 0) *d select(fileIndex == 1)' base.yaml patch.yaml
# Merge multiple files in sequence
yq eval-all '. as $item ireduce({}; . * $item)' base.yaml mid.yaml top.yaml
# Append to an array instead of replacing
yq eval-all 'select(fileIndex == 0).items + select(fileIndex == 1).items' a.yaml b.yaml
Helm-Style Value Overlays¶
# Helm merges values files in order (last wins)
helm template myapp ./chart \
-f values.yaml \
-f values-production.yaml \
--set image.tag=v2.1.0 # --set wins over all files
# Preview merged values
helm template myapp ./chart -f values.yaml -f values-prod.yaml --debug | head -50
# Kustomize patches (strategic merge)
# kustomization.yaml:
# patchesStrategicMerge:
# - increase-replicas.yaml
# - add-sidecar.yaml
Extracting Values from Kubernetes Manifests with yq¶
# Get all container images in a deployment
yq '.spec.template.spec.containers[].image' deployment.yaml
# Get all resource limits across all deployments
for f in k8s/deployments/*.yaml; do
echo "=== $f ==="
yq '.spec.template.spec.containers[] | {(.name): .resources.limits}' "$f"
done
# Find deployments without resource limits
for f in k8s/deployments/*.yaml; do
missing=$(yq '.spec.template.spec.containers[] | select(.resources.limits == null) | .name' "$f")
[ -n "$missing" ] && echo "$f: $missing missing limits"
done
# Extract all unique labels across manifests
find k8s/ -name '*.yaml' -exec yq '.metadata.labels // {} | keys | .[]' {} + | sort -u
# Get the image tag from a running cluster and compare to manifests
kubectl get deployments -o json | jq -r '.items[] | "\(.metadata.name): \(.spec.template.spec.containers[0].image)"' > running.txt
for f in k8s/deployments/*.yaml; do
name=$(yq '.metadata.name' "$f")
image=$(yq '.spec.template.spec.containers[0].image' "$f")
echo "$name: $image"
done > manifests.txt
diff running.txt manifests.txt
JSON Log Parsing with jq¶
Modern services emit structured JSON logs. jq is essential for parsing them.
# Filter by log level
cat app.log | jq 'select(.level == "error")'
# Extract timestamp and message only
cat app.log | jq -r '"\(.timestamp) \(.level) \(.message)"'
# Count by level
cat app.log | jq -s 'group_by(.level) | map({level: .[0].level, count: length})'
# Find slow requests (response time > 1000ms)
cat app.log | jq 'select(.response_time_ms > 1000) | {path: .request_path, ms: .response_time_ms}'
# Time-range filter
cat app.log | jq 'select(.timestamp >= "2024-01-15T10:00:00" and .timestamp <= "2024-01-15T11:00:00")'
# Top 10 slowest endpoints
cat app.log | jq -s 'map(select(.response_time_ms)) | sort_by(-.response_time_ms) | .[0:10] | .[] | {path: .request_path, ms: .response_time_ms}'
# Error rate per minute (assuming ISO timestamps)
cat app.log | jq -r '.timestamp[0:16]' | sort | uniq -c | sort -rn | head
# Parse Docker container logs
docker logs mycontainer 2>&1 | jq -R 'fromjson? // {raw: .}' | jq 'select(.level == "error")'
# Parse kubectl logs (JSON format)
kubectl logs deployment/myapp --since=1h | jq -R 'fromjson?' | jq 'select(.status >= 500)'
Validating Against JSON Schema¶
# Install ajv-cli (the standard JSON Schema validator)
npm install -g ajv-cli
# Validate a JSON file
ajv validate -s schema.json -d config.json
# Validate YAML (convert first)
yq -o json values.yaml | ajv validate -s values-schema.json -d -
# Python validation (no npm needed)
pip install jsonschema
python3 -c "
import json, jsonschema, sys
schema = json.load(open(sys.argv[1]))
data = json.load(open(sys.argv[2]))
jsonschema.validate(data, schema)
print('Valid')
" schema.json config.json
# Validate Helm values against a schema
# values.schema.json in your chart directory (Helm 3.1+ reads it automatically)
helm lint ./mychart -f values-prod.yaml
Config Drift Detection¶
Detect when deployed config diverges from what's in git.
# Compare live k8s configmap to git source
kubectl get configmap myapp-config -o yaml | yq '.data' > /tmp/live.yaml
yq '.data' k8s/configmap.yaml > /tmp/git.yaml
diff /tmp/live.yaml /tmp/git.yaml
# Compare all configmaps in a namespace
for cm in $(kubectl get configmaps -o name); do
name=$(echo "$cm" | cut -d/ -f2)
gitfile="k8s/configmaps/${name}.yaml"
[ -f "$gitfile" ] || { echo "DRIFT: $name exists in cluster but not in git"; continue; }
live=$(kubectl get "$cm" -o json | jq -S '.data')
git=$(yq -o json '.data' "$gitfile" | jq -S '.')
if [ "$live" != "$git" ]; then
echo "DRIFT: $name differs"
diff <(echo "$live") <(echo "$git")
fi
done
# Compare Helm release values to source values file
helm get values myrelease -o json | jq -S '.' > /tmp/deployed.json
yq -o json values-prod.yaml | jq -S '.' > /tmp/source.json
diff /tmp/deployed.json /tmp/source.json
# Full diff of rendered manifests vs live state
helm template myrelease ./chart -f values-prod.yaml > /tmp/rendered.yaml
kubectl diff -f /tmp/rendered.yaml # kubectl diff shows what would change
# Detect env var drift in running containers
kubectl get deployment myapp -o json | \
jq '.spec.template.spec.containers[0].env | sort_by(.name)' > /tmp/live-env.json
yq -o json '.spec.template.spec.containers[0].env' deployment.yaml | \
jq 'sort_by(.name)' > /tmp/git-env.json
diff /tmp/live-env.json /tmp/git-env.json