--- tags: - observability - l1 - flashcard-deck - tempo --- [Portal](../../../../library/portal/index.md) | **Level:** [L1: Foundations](../../../../library/portal/levels.md) | **Topics:** [Tempo](../../../../library/portal/topics.md) | **Domain:** Observability id category difficulty tags question answer source_path tempo/a1d2e3f4c5b6 tempo easy tempo, traces, observability What question do distributed traces answer that metrics and logs cannot? "Where in the system is a problem happening?" — traces show request flow across services, revealing which service or hop introduced latency or errors."\n\nRemember: "Tempo = Grafana's tracing backend." It stores traces in object storage (S3/GCS), making it cost-effective at scale.\n\nSee also: Tempo pairs with Grafana for visualization, Loki for logs, and Mimir for metrics. training/library/topics/observability-deep-dive/primer.md tempo/b2e3f4a5d6c7 tempo easy tempo, spans What is a span in distributed tracing? A span represents a single unit of work within a trace, with a name, start time, duration, and parent span reference. A trace is a tree of spans showing the full request lifecycle.\n\nRemember: "Trace = tree of spans." Each span has a parent, start time, duration, and metadata. The root span represents the entire request. training/library/topics/observability-deep-dive/primer.md tempo/c3f4a5b6e7d8 tempo easy tempo, trace-id What is a trace ID and how is it used? A trace ID is a globally unique identifier that ties all spans of a single request together across services. It is propagated in HTTP headers (e.g., traceparent) so each service can attach its spans to the same trace. training/library/topics/observability-deep-dive/primer.md tempo/d4a5b6c7f8e9 tempo medium tempo, architecture How does Tempo differ from Jaeger in its storage approach? Tempo stores traces in object storage (S3, GCS) without requiring a separate indexing database, making it cheaper and simpler to operate at scale. Jaeger typically requires Elasticsearch or Cassandra for indexing.\n\nRemember: "Trace ID = correlation key." Propagate it in HTTP headers (traceparent in W3C format) so all services link their spans to the same trace.\n\nGotcha: If any service drops the trace header, you get a broken trace. training/library/topics/observability-deep-dive/primer.md tempo/e5b6c7d8a9f0 tempo medium tempo, sampling What is trace sampling and why is it necessary? Sampling means only collecting a fraction of traces (e.g., 1% or 10%). It is necessary because capturing every trace at high throughput generates enormous volumes of data. Head-based sampling decides at the start; tail-based sampling decides after the trace completes (keeping only interesting traces). training/library/topics/observability-deep-dive/primer.md tempo/f6c7d8e9b0a1 tempo medium tempo, traceql What is TraceQL and what does it enable? TraceQL is Tempo's query language for searching traces by span attributes, duration, status, and resource fields. It allows queries like { span.http.status_code >= 500 && duration > 2s } to find slow, erroring requests. training/library/topics/observability-deep-dive/primer.md tempo/a7d8e9f0c1b2 tempo medium tempo, grafana, integration How does Tempo integrate with Grafana and Loki for end-to-end observability? Grafana links metrics, logs, and traces: a Prometheus alert can link to Loki logs via labels, and Loki log lines containing trace IDs become clickable links to Tempo traces, enabling drill-down from symptom to root cause. training/library/topics/observability-deep-dive/primer.md tempo/b8e9f0a1d2c3 tempo hard tempo, architecture, components What are the main components in Tempo's architecture? Distributor (receives spans from instrumented apps), Ingester (batches and writes to backend), Compactor (merges blocks in object storage), Querier (reads traces from storage), and Query Frontend (caches and splits queries).\n\nRemember: "Sampling reduces volume, not visibility." Head sampling decides at trace start; tail sampling decides after trace completes (keeps errors/slow traces).\n\nGotcha: Head sampling may drop interesting traces. Tail sampling is better but more complex. training/library/topics/observability-deep-dive/primer.md tempo/c9f0a1b2e3d4 tempo hard tempo, sampling, tail-based What is the advantage of tail-based sampling over head-based sampling? Tail-based sampling makes the keep/drop decision after the trace is complete, so it can retain traces with errors, high latency, or other interesting attributes. Head-based sampling decides at the start and may discard important traces by chance. training/library/topics/observability-deep-dive/primer.md tempo/d0a1b2c3f4e5 tempo hard tempo, exemplars, metrics How do exemplars bridge the gap between metrics and traces? Exemplars attach a trace ID to a specific metric data point (e.g., a histogram bucket observation). In Grafana, clicking an exemplar on a latency graph jumps directly to the Tempo trace for that request, connecting aggregate metrics to individual request detail. training/library/topics/observability-deep-dive/primer.md --- ## Wiki Navigation ### Related Content - [Observability Architecture](../../../../library/guides/observability.md) (Reference, L2) — Tempo - [Observability Deep Dive](../../../../library/topics/observability-deep-dive/index.md) (Topic Pack, L2) — Tempo - [Runbook: Tempo No Traces](../../../../library/runbooks/observability/tempo_no_traces.md) (Runbook, L2) — Tempo - [Skillcheck: Observability](../../../../library/skillchecks/observability.skillcheck.md) (Assessment, L2) — Tempo - [Track: Observability](../../../../library/curriculum/tracks/observability.md) (Reference, L2) — Tempo