--- tags: - linux - l1 - flashcard-deck - tracing --- [Portal](../../../../library/portal/index.md) | **Level:** [L1: Foundations](../../../../library/portal/levels.md) | **Topics:** [Tracing](../../../../library/portal/topics.md) | **Domain:** Linux id category difficulty tags question answer source_path tracing/a1f2b3c4d5e6 tracing easy tracing, concepts, basics What is a trace in distributed tracing? A trace represents the full journey of a request through a distributed system. It is a directed acyclic graph of spans, showing how the request flows across services.\n\nRemember: "Trace = end-to-end request journey. Span = one unit of work within a trace." A trace is a tree of spans. training/library/topics/tracing/primer.md tracing/b2a3c4d5e6f7 tracing easy tracing, spans What is a span, and what data does it contain? A span represents a single unit of work within a trace. It has a name, start time, duration, parent span reference, key-value attributes (tags), and timestamped events.\n\nRemember: "Trace context propagation = passing trace IDs across service boundaries." Without propagation, you get disconnected spans. training/library/topics/tracing/primer.md tracing/c3b4d5e6f7a8 tracing easy tracing, propagation Why must trace context be propagated between services? Without propagation, each service creates isolated traces with no way to connect them. Propagating trace and span IDs in request headers (e.g., W3C traceparent) links all spans into a single end-to-end trace.\n\nRemember: "OpenTelemetry (OTel) = vendor-neutral observability framework." It handles traces, metrics, and logs with one SDK. training/library/topics/tracing/primer.md tracing/d4c5e6f7a8b9 tracing medium tracing, sampling What is the difference between head-based and tail-based sampling? Head-based sampling decides whether to trace at request start (e.g., sample 10%). Tail-based sampling decides after the trace completes, allowing you to keep errors and slow traces. Head is simpler; tail catches anomalies. training/library/topics/tracing/primer.md tracing/e5d6f7a8b9c0 tracing medium tracing, w3c, headers What information does the W3C traceparent header contain? It contains version (2 hex), trace-id (32 hex), parent span-id (16 hex), and trace flags (2 hex, where 01 = sampled). Example: 00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01.\n\nRemember: "Sampling controls trace volume." Head sampling = decide at start, Tail sampling = decide at end (keeps errors/slow). 100% sampling crushes storage. training/library/topics/tracing/primer.md tracing/f6e7a8b9c0d1 tracing medium tracing, opentelemetry What is the role of the OpenTelemetry Collector? The OTel Collector receives telemetry data (traces, metrics, logs) from instrumented applications, processes it (batching, filtering, enrichment), and exports it to one or more backends like Jaeger, Tempo, or Datadog.\n\nRemember: "Span attributes = structured metadata." Add business context: user_id, order_id, region. training/library/topics/tracing/primer.md tracing/a7f8b9c0d1e2 tracing medium tracing, correlation How do you correlate traces with logs? Inject the trace_id and span_id into log lines. This allows jumping from a log entry to the full trace in the tracing UI, connecting structured logs to the request's end-to-end journey. training/library/topics/tracing/primer.md tracing/b8a9c0d1e2f3 tracing hard tracing, backends How does Grafana Tempo differ from Jaeger in its storage approach? Tempo uses object storage (S3, GCS) without indexing — traces are retrieved by trace ID directly or discovered through logs/metrics. Jaeger indexes spans for searchability. Tempo is cheaper at scale but requires trace ID discovery from other signals. training/library/topics/tracing/primer.md tracing/c9b0d1e2f3a4 tracing hard tracing, pitfalls What happens when one service in a call chain is not instrumented for tracing? The trace chain breaks at that service. Downstream spans cannot be linked to upstream spans, resulting in disconnected trace fragments. This is the most common tracing deployment failure.\n\nRemember: "Three pillars of observability: Logs, Metrics, Traces." Logs = events, Metrics = aggregates, Traces = request flows. training/library/topics/tracing/primer.md tracing/d0c1e2f3a4b5 tracing hard tracing, attributes, security What should you never include in span attributes, and why? Never include PII (personal data) or secrets in span attributes. Trace data is stored in backends accessible to many engineers and may be retained for weeks. Sensitive data in spans creates compliance and security exposure. training/library/topics/tracing/primer.md tracing/e1d2f3a4b5c6 tracing medium tracing, instrumentation, setup What is the difference between automatic and manual instrumentation in OpenTelemetry? Automatic instrumentation uses agents or library hooks to create spans for known frameworks (HTTP, DB, gRPC) with zero code changes. Manual instrumentation uses the SDK to create custom spans for business logic. \nBest practice: use auto-instrumentation as a baseline, add manual spans for critical business operations. tracing/f2e3a4b5c6d7 tracing medium tracing, baggage, propagation What is OpenTelemetry Baggage, and how does it differ from span attributes? Baggage is key-value data propagated across service boundaries via headers (like trace context). Unlike span attributes (local to one span), baggage travels with the request. Use it for cross-cutting concerns like tenant-id or feature-flag, but keep it small — every downstream service receives it. tracing/a3f4b5c6d7e8 tracing hard tracing, async, queues How do you maintain trace context across asynchronous message queues? Inject the trace context (traceparent header) into the message metadata/headers when producing. On the consumer side, extract the context and create a new span linked to the producer span. This creates a causal link even though the spans are not parent-child (use Span Links in OTel). tracing/b4a5c6d7e8f9 tracing hard tracing, sampling, cost How do you optimize tracing costs without losing visibility into errors? Use tail-based sampling to keep 100% of error and high-latency traces while sampling routine traffic at 1-10%. Alternatively, use head-based sampling with a rule engine: always sample traces with specific headers (debug, canary), sample the rest probabilistically. Monitor the traces-per-second rate to stay within budget. tracing/c5b6d7e8f9a0 tracing medium tracing, debugging, orphans What causes orphaned spans, and how do you debug them? Orphaned spans have no parent and cannot be joined to a trace. Common causes: context not propagated (missing middleware), async boundary drops context, service restart loses in-flight context, or clock skew makes spans appear disconnected. Debug by checking: \n1) propagation headers in requests, \n2) instrumentation gaps, \n3) collector pipeline for dropped spans. tracing/d6c7e8f9a0b1 tracing easy tracing, observability, correlation How do traces complement metrics and logs in the three pillars of observability? Metrics show that something is wrong (high latency, error rate spike). Logs show what happened in a single service. Traces show why by revealing the full request path across services, exposing which service introduced latency or errors. Use metrics to detect, traces to diagnose, logs for detail. tracing/897b80b04e7f tracing medium tracing, span-links, batch When should you use span links instead of parent-child relationships? Span links connect causally related spans that are not in a direct parent-child hierarchy — for example, a batch job processing items from a queue where each item originated in a different trace. The link preserves the connection without forcing all items into one trace tree. training/library/topics/tracing/primer.md tracing/813a8c4785eb tracing hard tracing, otel-collector, pipeline What are the three pipeline stages in an OpenTelemetry Collector and why separate them? Receivers (accept data in various formats: OTLP, Jaeger, Zipkin), Processors (transform, filter, batch, sample), Exporters (send to backends: Tempo, Jaeger, OTLP). Separation lets you receive in one format, enrich/filter centrally, and export to multiple backends without changing instrumentation. training/library/topics/tracing/primer.md tracing/41fc6001b1a8 tracing hard tracing, service-mesh, envoy, sidecar How does a service mesh like Istio inject trace context without application changes? Envoy sidecars automatically generate spans for inbound/outbound requests and propagate trace headers (W3C traceparent or B3). The application only needs to forward the incoming trace headers on outbound calls. Without header forwarding, traces break into disconnected segments per hop. training/library/topics/tracing/primer.md tracing/f66e8c39af00 tracing medium tracing, critical-path, bottleneck What is critical path analysis in distributed tracing and how does it identify bottlenecks? The critical path is the longest chain of sequential spans from trace start to end. Spans on the critical path directly add to total latency; parallel spans off the critical path do not. Optimizing the longest span on the critical path yields the biggest latency reduction. training/library/topics/tracing/primer.md --- ## Wiki Navigation ### Related Content - [OpenTelemetry](../../../../library/topics/opentelemetry/index.md) (Topic Pack, L2) — Tracing - [Tracing](../../../../library/topics/tracing/index.md) (Topic Pack, L1) — Tracing - [perf Profiling](../../../../library/topics/perf-profiling/index.md) (Topic Pack, L2) — Tracing - [strace](../../../../library/topics/strace/index.md) (Topic Pack, L1) — Tracing