--- tags: - devops - l1 - flashcard-deck - chaos-engineering --- [Portal](../../../../library/portal/index.md) | **Level:** [L1: Foundations](../../../../library/portal/levels.md) | **Topics:** [Chaos Engineering](../../../../library/portal/topics.md) | **Domain:** DevOps & Tooling id category difficulty tags question answer source_path chaos-engineering/25eda09f2b57 chaos-engineering medium chaos-engineering, control-flow, kubernetes, text-processing Cite a few tools used to operate Chaos exercises - AWS Fault Injection Simulator: inject failures in AWS resources\n- Azure Chaos Studio: inject failures in Azure resources\n- Chaos Monkey: one of the most famous tools to orchestrate Chaos on diverse Cloud providers\n- Litmus - A Framework for Kubernetes \n- Chaos Mesh: for Cloud Kubernetes platforms\n\n\nSee an extensive list [here](https://github.com/dastergon/awesome-chaos-engineering)\n\nRemember: the goal of chaos engineering is confidence in your system's resilience, not finding bugs (though that's a welcome side effect). Run experiments regularly, not just once. projects/knowledge/interview/chaos-engineering/003-cite-a-few-tools-used-to-operate-chaos-exercises.txt chaos-engineering/cfff839aa4ec chaos-engineering easy chaos-engineering, networking, processes What is Chaos Engineering? "[Wikipedia](https://en.wikipedia.org/wiki/Chaos_engineering): ""Chaos Engineering is the discipline of experimenting on a system in order to build confidence in the system's capability to withstand turbulent conditions in production.""\n\n[TechTarget](https://www.techtarget.com/searchitoperations/definition/chaos-engineering): ""Chaos engineering is the process of testing a distributed computing system to ensure that it can withstand unexpected disruptions.""\n\nRemember: chaos engineering = 'break things on purpose to find weaknesses before they find you.' Proactive, not reactive.\n\nName origin: Netflix coined the term with Chaos Monkey (2011) — randomly kills EC2 instances in production to test resilience." projects/knowledge/interview/chaos-engineering/001-what-is-chaos-engineering.txt chaos-engineering/e3bd4b217fe5 chaos-engineering medium chaos-engineering, control-flow, networking, processes What's a typical Chaos Engineering workflow? According to [Gremlin](https://gremlin.com) there are three steps:\n\n1. Planning an experiment where you design and choose a scenario in which your system should fail to operate properly\n2. You execute the smallest possible experiment to test your theory\n3. If nothing goes wrong, you scale your experiment and make the blast radius bigger. If your system breaks, you better understand why and start dealing with it\n\nThe process then repeats itself either with same scenario or a new one.\n\nRemember: the goal of chaos engineering is confidence in your system's resilience, not finding bugs (though that's a welcome side effect). Run experiments regularly, not just once. projects/knowledge/interview/chaos-engineering/002-whats-a-typical-chaos-engineering-workflow.txt chaos-engineering/train-chaos-a chaos-engineering medium chaos-engineering, kubernetes, training What safety controls should chaos scripts have? 1) Namespace scoping (only approved namespaces); 2) Dry-run mode by default; 3) Explicit --yes flag for destructive operations; 4) Built-in restore/cleanup; 5) Never operate on kube-system without explicit override; 6) Reversible changes only.\n\nRemember: the goal of chaos engineering is confidence in your system's resilience, not finding bugs (though that's a welcome side effect). Run experiments regularly, not just once. training/interactive/chaos/README.md chaos-engineering/train-chaos-b chaos-engineering medium chaos-engineering, kubernetes, training Name 5 safe chaos experiments for a Kubernetes namespace. 1) Kill pods (deployment controller recreates them); 2) Break readiness probe (pod drops from service); 3) CPU/memory stress pods (test resource limits); 4) Apply deny-all NetworkPolicy (test network resilience); 5) Scale deployment to zero then restore (test recovery time).\n\nRemember: the goal of chaos engineering is confidence in your system's resilience, not finding bugs (though that's a welcome side effect). Run experiments regularly, not just once. training/interactive/chaos/README.md chaos-engineering/train-chaos-c chaos-engineering hard chaos-engineering, kubernetes, training How do you introduce chaos safely in a production-like environment? Start small: 1) Namespace-scoped only; 2) One failure at a time; 3) Have runbooks ready; 4) Monitor during experiment; 5) Automated rollback on unexpected impact; 6) GameDay format with the team present; 7) Document findings and improve resilience. Never surprise people.\n\nRemember: the goal of chaos engineering is confidence in your system's resilience, not finding bugs (though that's a welcome side effect). Run experiments regularly, not just once. training/interactive/chaos/README.md chaos-engineering/ce-litmus-vs-gremlin chaos-engineering medium chaos-engineering, tools, comparison Compare Litmus, Gremlin, and Chaos Monkey as chaos engineering tools. Chaos Monkey (Netflix): random instance termination in AWS, pioneered the discipline.\nGremlin: commercial SaaS platform with a UI, supports CPU/memory/network/state attacks, rollback controls, and team collaboration.\nLitmus: CNCF open-source project, Kubernetes-native, uses ChaosEngine CRDs, supports litmus experiments from a hub.\nKey difference: Chaos Monkey is narrow (kill instances), Gremlin is broad and commercial, Litmus is K8s-native and open-source.\n\nSee also: Chaos Monkey (Netflix), LitmusChaos (CNCF), Gremlin (SaaS), chaos-mesh (K8s-native), Pumba (container chaos). training/interactive/knowledge/data/cards/chaos-engineering.tsv chaos-engineering/ce-steady-state chaos-engineering easy chaos-engineering, concepts, fundamentals What is a steady-state hypothesis in chaos engineering? "A steady-state hypothesis defines what ""normal"" looks like for your system before an experiment begins. It is a measurable assertion (e.g., p99 latency < 200ms, error rate < 0.1%) that you expect to remain true during the experiment. If the steady state breaks, the experiment has found a weakness.\n\nRemember: chaos experiments follow the scientific method: hypothesis -> inject fault -> observe -> conclude. 'We believe killing one node will not increase error rate above 0.1%.'" training/interactive/knowledge/data/cards/chaos-engineering.tsv chaos-engineering/ce-blast-radius chaos-engineering medium chaos-engineering, safety, blast-radius What is blast radius in chaos engineering and how do you control it? Blast radius is the scope of impact an experiment can have. Control it by: 1) Start in non-production environments; 2) Target a single service or pod first; 3) Use namespace scoping; 4) Set duration limits and auto-abort conditions; 5) Run during low-traffic windows; 6) Have a kill switch to stop the experiment instantly; 7) Gradually expand scope only after confidence grows.\n\nRemember: start with the smallest blast radius (one pod, one instance) and expand. Never go straight to 'kill an AZ' without validating smaller failures first. training/interactive/knowledge/data/cards/chaos-engineering.tsv chaos-engineering/ce-gameday chaos-engineering medium chaos-engineering, practice, gameday What is a GameDay and how do you plan one? A GameDay is a scheduled chaos exercise where teams inject failures into a system while observing the impact. Planning steps: 1) Define scope and objectives; 2) Select experiments (start simple); 3) Brief all participants on the plan and rollback procedures; 4) Establish communication channels; 5) Run experiments while monitoring dashboards; 6) Document observations in real time; 7) Hold a blameless retrospective; 8) Create follow-up action items.\n\nRemember: the goal of chaos engineering is confidence in your system's resilience, not finding bugs (though that's a welcome side effect). Run experiments regularly, not just once. training/interactive/knowledge/data/cards/chaos-engineering.tsv chaos-engineering/ce-cicd-chaos chaos-engineering hard chaos-engineering, ci-cd, automation How can chaos experiments be integrated into a CI/CD pipeline? Run lightweight chaos tests as a pipeline stage after deployment to a staging environment. Approach: 1) Deploy to staging; 2) Verify steady state via health checks; 3) Inject a controlled failure (e.g., kill a pod, add latency); 4) Assert steady-state hypothesis holds; 5) Fail the pipeline if assertions break. Tools like Litmus, Gremlin, or Chaos Mesh provide CLI/API interfaces suitable for pipeline integration. Keep experiments short and auto-revert.\n\nRemember: the goal of chaos engineering is confidence in your system's resilience, not finding bugs (though that's a welcome side effect). Run experiments regularly, not just once. training/interactive/knowledge/data/cards/chaos-engineering.tsv chaos-engineering/ce-rollback-triggers chaos-engineering medium chaos-engineering, safety, rollback What rollback triggers should chaos experiments have? Auto-abort when: 1) Error rate exceeds a threshold (e.g., 5x baseline); 2) Latency spikes beyond SLO; 3) A dependent service becomes unreachable; 4) Customer-facing impact is detected; 5) The experiment duration exceeds the planned window; 6) An operator manually hits the kill switch. Always define these triggers before the experiment starts.\n\nRemember: the goal of chaos engineering is confidence in your system's resilience, not finding bugs (though that's a welcome side effect). Run experiments regularly, not just once. training/interactive/knowledge/data/cards/chaos-engineering.tsv chaos-engineering/ce-observability chaos-engineering medium chaos-engineering, observability, monitoring Why is observability critical during chaos experiments? Without observability you cannot measure impact or validate your steady-state hypothesis. You need: 1) Real-time dashboards showing key metrics (latency, error rate, throughput); 2) Distributed tracing to see how failures propagate; 3) Log aggregation to capture error details; 4) Alerting that fires during the experiment to validate alert coverage. Chaos experiments often reveal observability gaps themselves.\n\nRemember: the goal of chaos engineering is confidence in your system's resilience, not finding bugs (though that's a welcome side effect). Run experiments regularly, not just once. training/interactive/knowledge/data/cards/chaos-engineering.tsv chaos-engineering/ce-failure-types chaos-engineering easy chaos-engineering, concepts, failure-types Name the main categories of failure injection in chaos engineering. Infrastructure: instance/node termination, disk fill, clock skew.\nNetwork: latency injection, packet loss, DNS failure, partition.\nApplication: process kill, memory pressure, CPU stress, exception injection.\nDependency: external service unavailability, slow responses, certificate expiry.\nState: data corruption, cache invalidation, queue backup.\n\nRemember: the goal of chaos engineering is confidence in your system's resilience, not finding bugs (though that's a welcome side effect). Run experiments regularly, not just once. training/interactive/knowledge/data/cards/chaos-engineering.tsv chaos-engineering/ce-principles chaos-engineering easy chaos-engineering, concepts, principles List the core principles of chaos engineering. 1) Build a hypothesis around steady-state behavior; 2) Vary real-world events (simulate realistic failures); 3) Run experiments in production when possible; 4) Automate experiments to run continuously; 5) Minimize blast radius. These principles come from the Chaos Engineering manifesto (principlesofchaos.org).\n\nRemember: the goal of chaos engineering is confidence in your system's resilience, not finding bugs (though that's a welcome side effect). Run experiments regularly, not just once. training/interactive/knowledge/data/cards/chaos-engineering.tsv chaos-engineering/ce-resilience-vs-chaos chaos-engineering easy chaos-engineering, concepts, resilience What is the difference between resilience testing and chaos engineering? "Resilience testing verifies known failure modes with expected outcomes (like a test suite). Chaos engineering explores unknown failure modes by injecting real-world turbulence and observing what happens. Resilience testing asks ""does our retry logic work?"" while chaos engineering asks ""what breaks when the database is slow for 30 seconds?""\n\nRemember: the goal of chaos engineering is confidence in your system's resilience, not finding bugs (though that's a welcome side effect). Run experiments regularly, not just once." training/interactive/knowledge/data/cards/chaos-engineering.tsv chaos-engineering/ce-aws-fis chaos-engineering hard chaos-engineering, tools, aws How does AWS Fault Injection Simulator (FIS) work? FIS is a managed service for running chaos experiments on AWS resources. You define an experiment template specifying: 1) Targets (EC2 instances, ECS tasks, RDS clusters); 2) Actions (stop instances, inject API errors, throttle network); 3) Stop conditions (CloudWatch alarm thresholds). FIS integrates with IAM for permissions and CloudWatch for monitoring. It supports gradual rollout and automatic stop on alarm.\n\nRemember: the goal of chaos engineering is confidence in your system's resilience, not finding bugs (though that's a welcome side effect). Run experiments regularly, not just once. training/interactive/knowledge/data/cards/chaos-engineering.tsv chaos-engineering/ce-chaos-mesh chaos-engineering hard chaos-engineering, tools, kubernetes What is Chaos Mesh and how does it inject faults in Kubernetes? Chaos Mesh is a CNCF sandbox project for cloud-native chaos engineering on Kubernetes. It uses CRDs to define experiments (PodChaos, NetworkChaos, StressChaos, IOChaos). It injects faults by manipulating pod containers via sidecar injection or privileged DaemonSets. Features include a web dashboard, RBAC integration, scheduled experiments, and workflow orchestration for multi-step chaos scenarios.\n\nRemember: chaos engineering = 'break things on purpose to find weaknesses before they find you.' Proactive, not reactive.\n\nName origin: Netflix coined the term with Chaos Monkey (2011) — randomly kills EC2 instances in production to test resilience. training/interactive/knowledge/data/cards/chaos-engineering.tsv chaos-engineering/076d156c3250 chaos-engineering medium chaos-engineering, observability, detection Why must you validate observability before running chaos experiments? If dashboards and alerts cannot detect the failure you are about to inject, the experiment is wasted — you learn nothing about detection capability. Run a pre-check: verify the relevant SLI dashboard shows the baseline, alerts are active, and on-call knows the experiment is happening.\n\nRemember: the goal of chaos engineering is confidence in your system's resilience, not finding bugs (though that's a welcome side effect). Run experiments regularly, not just once. training/library/topics/chaos-engineering/primer.md chaos-engineering/70a90889bc3a chaos-engineering hard chaos-engineering, maturity, progression What does a chaos engineering maturity progression look like? Level 1: Kill single processes, observe recovery. Level 2: Network faults (latency, partition) between services. Level 3: Dependency failures (database, cache, DNS). Level 4: Multi-AZ/region failures. Level 5: Automated chaos in CI/CD with auto-rollback. Each level requires stronger observability and blast-radius controls before advancing.\n\nRemember: the goal of chaos engineering is confidence in your system's resilience, not finding bugs (though that's a welcome side effect). Run experiments regularly, not just once. training/library/topics/chaos-engineering/primer.md --- ## Wiki Navigation ### Related Content - [Chaos Engineering & Fault Injection](../../../../library/topics/chaos-engineering/index.md) (Topic Pack, L2) — Chaos Engineering