Correlating Service Telemetry Across a Fragmented Operations Estate

An anonymous financial-services engagement connecting metrics, traces and logs through shared context, with staged instrumentation and operational ownership.

Operators needed to follow a request across tools

A financial-services platform had separate logging systems, inconsistent metrics and limited cross-service tracing. During an incident, engineers needed to find the relevant evidence across those tools before they could investigate the request.

Ampity's work combined a shared telemetry model, OpenTelemetry instrumentation and a unified query interface. The design named Prometheus and Thanos for metrics and longer-term storage, Loki for logs, and Grafana for investigation. The operating question was whether an authorized engineer could connect a service symptom to the request and diagnostic events that explain it.

Consolidating tools leaves a correlation contract to define

Commercial APM, retaining a logging-and-metrics arrangement and a broader open-source stack were alternatives. The chosen direction kept more telemetry-platform responsibility with the team. It also required instrumentation, configuration, retention, upgrade and incident ownership.

A single interface does not make separately collected signals relate correctly. Choose stable service and environment identities, define where request context crosses process boundaries and decide what evidence an investigation needs. The case leaves the deployed tracing backend unspecified; the correlation problem should not depend on silently choosing one.

Connect signals without making every request a metric label

OpenTelemetry was used as the instrumentation approach, with shared context intended to connect traces and diagnostic logs. That requires an effective propagation and collection path, not merely a field with the same name in several tools.

Metrics summarize selected measurements. Where the deployed stack supports exemplars, they can connect a particular measurement to trace context. That is different from attaching an unbounded request identifier to every metric series. Review the actual exemplar, sampling and backend behavior before promising an alert-to-trace link.

Treat trace context as correlation data, not tenant authorization. Access, sensitive fields and retention need their own controls. Routine diagnostic logs also do not become immutable regulatory audit records merely because they appear in the same dashboard.

Introduce instrumentation without removing the existing response path

The rollout approach combined core platform setup, shared instrumentation for Java and Go, onboarding of critical services, then self-service documentation and training. Existing alert coverage and gradual adoption remained constraints across Kubernetes, virtual machines, AWS and on-premises workloads.

For a comparable rollout, retain the alert path until its replacement has an owner and usable evidence. Instrumentation can add application work and telemetry volume; set a bound before expanding it. Missing traces can reflect collection or sampling gaps rather than an absence of failures.

Follow one permitted failure from signal to operator action

Begin with authorized read-only inspection of permitted instrumentation, query and alert definitions. This worksheet is proposed guidance, not an executed incident or sampling exercise from this engagement.

Any later exercise needs an authorized isolated environment, synthetic requests and markers, restricted test identities and non-delivering alert/export destinations. Prevent real customer payloads, production pages, external notifications and uncontrolled telemetry load. Stop on prohibited content, cross-tenant exposure or unexpected application pressure.

Restoring instrumentation or a query can change future evidence. It cannot recreate telemetry already dropped or recall sensitive data already exported. Preserve the known gap and assign any required data containment separately.

Scope without a resolution-time or savings claim

This case covers telemetry correlation, instrumentation and staged adoption. It does not publish measured incident-resolution improvements, operating savings, developer-satisfaction gains, complete trace coverage or an achieved compliance result.

Use one investigation to decide what is missing

Pick a permitted service operation and identify the first question an on-call engineer cannot answer from its existing signals. Add only the evidence needed for that question, with a named owner and handling boundary.

The observability operating playbook covers instrumentation, sampling and operator-response validation in depth. The telemetry-redaction review addresses where sensitive fields can be copied before the dashboard sees them.

Technical reference

OpenTelemetry explains context propagation and the optional trace context carried by metric exemplars. Prometheus's instrumentation guidance describes the resource cost of label sets. These mechanisms support a reviewed correlation design; they do not establish this engagement's actual sampling, backend, access or retention configuration.