Observability and Monitoring: Build an Answerable Investigation

Connect user-impact signals to traces, structured logs and release context. A worked latency investigation shows what the evidence establishes and where it stops.

An alert can tell you that report generation is slower. The next question is whether the delay comes from database work, an external service, waiting for capacity or a change in the requests being served.

Monitoring and investigation should use the same operational evidence. Metrics can identify the affected population and when it changed. Traces can expose the work along a sampled request. Structured logs can record decisions that a duration alone cannot explain. A dashboard, query language or vendor choice does not settle whether those questions are answerable.

This guide helps a service team build that investigation path. It does not require recording every event or promise that existing telemetry will answer every future question.

Start with an operational question

Choose one user-visible operation and define success, failure and acceptable delay. For an asynchronous job, measure acceptance and completion separately. A healthy submission endpoint can hide a queue in which jobs wait indefinitely.

Write a few questions the on-call engineer needs to answer. Which operation is affected? Did it start with a release? Does the delay occur before work begins or during a dependency call? Are failures confined to a bounded region or request type?

| Evidence | Useful question | | --- | --- | | Rates and latency distributions | How widespread is the change, and when did it begin? | | A representative trace | Where did this request spend time? | | Structured application events | Which branch, retry or fallback did the code take? | | Release and configuration identity | Which version and operating settings handled it? | | Queue age and completion state | Is work progressing after acceptance? |

OpenTelemetry's signal overview describes metrics, traces and logs as complementary telemetry. Instrumentation and storage provide the evidence; the team still has to define what it needs to investigate.

Do not make raw customer identity the first answer to every diagnostic question. A bounded request category, dependency name or deployment version may explain the failure without identifying a person.

Keep distributions, not only averages

Metrics systems are not limited to averages. Prometheus supports histograms and summaries, with different aggregation and quantile behavior. Choose a representation that supports the queries and accuracy you need.

An average can conceal a small slow population. In an illustrative set of 100 requests, 99 take 100 ms and one takes 2,100 ms. The mean is 120 ms: (99 × 100 + 2,100) / 100. That figure does not describe the slow request.

Compare latency distributions, error counts and request volume for the same operation and time window. An apparent improvement after traffic disappears is not a recovery. For histograms, ensure bucket boundaries or native-histogram precision support the service's latency decision. Quantiles are estimates with representation-specific limitations.

Do not average instance-level percentile values to obtain a service percentile. Use the appropriate aggregation over the underlying distribution when supported. Record the query and filters with the incident evidence so another engineer can reproduce the comparison.

Worked investigation: a slow report request

Suppose a fictional report endpoint shows a latency change after a deployment. Error rate remains stable. A captured 900 ms trace contains the following sequential, non-overlapping work; these are teaching values, not production benchmarks.

| Segment | Observed duration | | --- | --- | | Prepare request | 40 ms | | Fetch report rows | 120 ms | | Partner enrichment call | 640 ms | | Encode response | 100 ms | | Total request | 900 ms |

The partner-call span accounts for 640 of the 900 ms, so it is a useful next investigation point. It does not prove that the partner's application is slow. The client span may include connection establishment, network delay, retries or proxy waiting. Confirm what the instrumentation measures.

Compare slow and ordinary requests from the same release. Check the attempt count, dependency endpoint class, payload-size band and connection behavior. Then compare with the earlier release under similar traffic. If retry count changed, inspect the effective SDK and proxy policies before increasing the timeout.

For concurrent or nested spans, do not add every duration. Parent spans include child time, and parallel spans overlap. Follow the critical path and distinguish waiting from active work. Missing spans and unsynchronized clocks can also make the visual timeline misleading.

A useful investigation record states the observation, competing explanations, next check and result. “Partner call is slow” is an observation. “New retry policy repeats a failed connection” is a hypothesis until configuration and attempt evidence support it.

Carry context across the boundaries you operate

Use context propagation to connect instrumented work. Confirm that context survives the actual ingress, service calls and messaging paths in your stack. Adding a tracing library to each service does not guarantee a connected trace.

For asynchronous work, record the job identifier and the relevant trace relationship. A later processing span may use a parent or link according to the messaging convention and execution model. Keep business identifiers and tracing identifiers distinct: a trace can end while a job remains pending.

Test an allowed request end to end, including a retry and an error path. Verify that the service name, version, environment and operation attributes are consistent. A missing context edge should appear as a coverage gap, not be mistaken for proof that downstream work did not happen.

Trace headers and baggage arrive from outside the trust boundary in some systems. Validate their use and do not treat them as authentication. Do not put secrets or unnecessary personal data into propagated baggage, which may travel to other services.

Some investigations will need new instrumentation. That is a legitimate improvement when the missing evidence is identified and collected safely. It is not evidence that the entire observability program has failed.

Choose dimensions without creating unbounded metrics

Prometheus naming guidance warns that each unique label combination creates a time series. User IDs, request IDs and raw URLs can grow without a useful bound.

For a simplified example, four methods, twelve route templates, three status classes and two regions permit 288 label combinations. Adding 100,000 distinct user IDs permits up to 28.8 million combinations before accounting for instances or histogram components. Actual combinations may be fewer, but the new dimension changes the capacity problem.

Use route templates instead of paths containing record IDs. Consider bounded error categories instead of arbitrary exception text. Keep request-level correlation in suitably controlled traces or logs, not as a label on every metric.

A high-cardinality trace attribute can be useful, but its cost, indexing behavior and privacy implications still need review. The appropriate decision depends on query need and backend behavior, not a rule that every identifier belongs everywhere.

Treat telemetry as data that needs protection

Collect the minimum fields required for the operating question. Exclude credentials, authorization headers, session tokens and raw request bodies by default. Review exceptions at the source, before the data enters queues, buffers or external storage.

When correlation requires an identifier, consider a scoped pseudonymous value with an approved mapping and lifetime. Pseudonymization does not make data anonymous. Predictable identifiers can remain guessable after hashing, a limitation described in OpenTelemetry's sensitive-data guidance.

Document who can query detailed events, where they are stored, how long they are retained and how deletion obligations apply. Aggregates and detailed traces may need different access and retention policies. Diagnostic convenience is not permission to retain every raw feature indefinitely.

Test redaction with synthetic sensitive values. Confirm they are absent from application output, collector buffers and the destination, including error paths. A processor configured in one pipeline does not prove that another export route is protected.

State what sampling can hide

Sampling trades collection volume against retained detail. Head sampling decides early; tail sampling can consider later trace information but requires buffering and an appropriate collection topology.

Neither approach gives a complete event history automatically. A rare failure may be missed, and a backend retaining disproportionately slow traces cannot by itself establish the population's error rate. Use appropriately collected metrics for population questions and selected traces for investigation.

Record sampling rules, drop counts, export failures and collector resource pressure. Test whether a chosen error case is retained, while acknowledging that one successful test does not establish coverage for all failures.

Budget collection by signal and query need. Reduce redundant fields or low-value events before assuming that keeping everything is the safest choice. Also test collector or destination failure: application behavior under telemetry backpressure should be understood.

Run an investigation rehearsal

Pick a safe test failure, such as a deliberately delayed test dependency, and ask an engineer unfamiliar with the implementation to investigate it.

| Check | Evidence to retain | | --- | --- | | User impact is visible | Operation, affected interval, volume and success or completion signal | | A representative request is traceable | Context crosses required boundaries or records a known gap | | Competing causes can be tested | Dependency timing, retries, release identity and decision events | | Sensitive values are excluded | Synthetic redaction test and access review | | Collection loss is visible | Sampling, dropped telemetry and export-failure signals | | The result is reproducible | Saved query, trace reference, hypothesis and verified outcome |

Use the production debugging guide for the investigation process once evidence is available.

For Ampity's CI/CD and observability service, bring the failed questions and the rehearsal record. That gives the review a concrete scope: instrumentation, context propagation, query design or operating ownership, with any monitoring coverage agreed separately.