Observability for Modern Platforms

Instrument a user journey, protect telemetry data, validate sampling and delivery, and connect reliability signals to tested operator actions.

trigger="Operators cannot explain a production failure, alerts are unreliable, or a new service needs an observability contract." owner="The service owner accountable for detecting and responding to user-impacting failures." participants={["Application engineer", "Telemetry platform owner", "On-call representative", "Product owner", "Security and data reviewers"]} prerequisites={[ "A defined user journey, known failure modes, and an inventory of its dependencies.", "Approved telemetry fields, access, retention, and an operating cost boundary.", "A representative environment for instrumentation, sampling, alert-delivery, and recovery tests." ]} outputs={[ "A versioned telemetry contract and measurable user-facing SLI.", "A validated collection and sampling path with coverage and loss evidence.", "Owned alerts, tested runbooks, and an instrumentation rollout and recovery record." ]} doneWhen={[ "An operator can follow a representative failed operation using authorized telemetry.", "SLI calculations and alerts behave correctly for success, failure, low traffic, and missing data.", "Sampling and collection limits are measured and visible to the operating team.", "Each retained or retired alert has a documented risk-coverage decision." ]} />

Start with an operator question

Choose one question the team cannot answer reliably. For example: which accepted exports missed their completion deadline, where did they wait, and what can an operator safely do next?

Trace that operation through the application, queue, worker, data store, and external dependency. Identify the evidence needed at each boundary. Some incidents will still require new instrumentation; that does not prove the organization is immature. Record the gap and add evidence that serves a concrete investigation.

This playbook covers instrumentation and operational response. Use observability cost optimization for a deeper cost review, and the enterprise observability strategy whitepaper for organization-wide planning.

1. Define the user signal and its denominator

The service and product owners agree what successful behavior means. A server returning a response can differ from a user completing a transaction or receiving a correct result.

Record the SLI's numerator, denominator, observation point, exclusions, and measurement window. For asynchronous work, acceptance and completion may need separate indicators. For latency, define which operations qualify and what duration is acceptable.

Use representative failure examples to check the calculation. What happens when the client times out, a retry succeeds, a job completes late, or telemetry is missing? Avoid counting retries as new successful business operations unless that matches the stated measure.

Synthetic checks can test a controlled journey, while real-user or server-side signals cover different parts of actual demand. Neither source automatically provides the complete truth. Document the blind spots and avoid exposing customer data in synthetic tests.

Work through an asynchronous export before choosing dashboards

Suppose the selected journey is a customer requesting a data export. The API can accept the request while the worker never completes it. The service owner therefore defines a completion indicator from a cohort of accepted logical jobs whose agreed deadline has elapsed. The numerator is the number that produced a valid, retrievable export before that deadline. The denominator includes every eligible job in that cohort, including failures and missing terminal states.

The owner defines cancellation and invalid-request rules before collecting results. A user cancellation that legitimately ends the obligation may be excluded under the documented policy. An operator canceling a stuck job must not quietly remove a service failure from the denominator. Retries retain the logical job identity so repeated execution cannot manufacture extra successful exports.

The application engineer emits accepted, started, completed, and failed events with a stable job reference and configuration version. The data owner determines how that reference can be stored safely. The completion record comes from the component that knows the artifact is valid and available, not from the queue acknowledgment alone. A reconciliation query finds accepted jobs with no terminal state after the deadline.

The acceptance test seeds an on-time success, a late success, a failure, a duplicate attempt, an authorized cancellation, and a missing completion event. The reviewer computes the expected indicator from that manifest and compares it with the production query. Keep the query revision and manifest with the telemetry contract. If observed results differ, fix the semantics before discussing an SLO target.

2. Write a telemetry contract

The application engineer specifies events, metrics, and spans that answer the chosen questions. Use stable operation names and consistent units. Automatic instrumentation can establish a baseline, but support varies by library and version.

| Signal | Use | Control to specify | | --- | --- | --- | | Metrics | Aggregate demand, errors, latency, saturation, backlog age | Bounded dimensions, units, histogram or counter semantics | | Logs | Discrete events and diagnostic context | Field allowlist, severity, redaction, retention | | Traces | Timing and relationships across instrumented operations | Context propagation, sampling, sensitive attributes | | Audit records | Attributable security or business actions | Separate access, integrity, retention, and review requirements |

Logs are not automatically immutable audit evidence. If an audit use case requires stronger controls, the security and data owners define them separately from routine diagnostic logging.

Preserve service name, deployed version, environment, and useful operation correlation. Test that identifiers connect the relevant evidence rather than assuming identical field names guarantee correlation.

Use this contract worksheet during implementation. Each row should name a source and an owner; “available in the dashboard” is not enough to reconstruct a missing signal.

| Field | Example for the export journey | Acceptance question | | --- | --- | --- | | Logical operation | One accepted export job | Do retries keep the same business identity? | | State observation | Artifact validated and access enabled | Can a failed upload look completed? | | Timing | Accepted time, deadline, completion time | Are clocks and late arrivals handled explicitly? | | Required dimensions | Service version, environment, bounded job class | Can arbitrary input create a new metric series? | | Investigation reference | Approved job correlation token in restricted logs | Can an authorized operator find the matching trace? | | Loss signal | Missing terminal-state count and exporter health | Can telemetry loss make the service appear healthy? |

The application owner reviews the contract with the on-call engineer using one real diagnostic question. Remove fields that do not support that question or an approved requirement. Add a field only when its meaning, source, allowed values, handling policy, and consumer are clear. This avoids collecting an expensive payload dump that still omits the state needed to explain failure.

3. Approve identifiers and cardinality before collection

The data reviewer classifies proposed fields and approves their purpose, destination, access, and retention. User and tenant identifiers can be sensitive or linkable. Hashing them does not automatically make them anonymous.

Do not log passwords, access tokens, session secrets, or unreviewed request bodies. Inspect what SDKs and exception handlers emit, including URLs, query parameters, headers, and database statements. Redact at the source where possible and add collector controls as another layer.

OpenTelemetry's sensitive-data guidance explains the implementer's responsibility to review instrumentation and minimize collected information. Treat exports to another telemetry backend as a data flow that also needs approval.

For metrics, estimate the number of series produced by combinations of labels and instances. Request IDs and unconstrained user IDs can create unbounded series growth. Prometheus instrumentation guidance describes the resource cost of additional label sets. Prefer controlled logs or traces for detailed investigation when appropriate.

Set a field and volume budget per workload. Verify tenant-scoped access to shared telemetry, including saved queries, exports, and support dashboards. A tenant filter in a chart is not necessarily an authorization boundary.

4. Instrument and test the complete operation

The application engineer starts with a representative path and adds manual spans or events where automatic instrumentation misses business state. Capture acceptance, completion, and failure reasons without copying unnecessary payloads.

Propagate trace context through supported HTTP, RPC, queue, and worker boundaries. For asynchronous or batch work, use the tracing model appropriate to the relationship, including links where applicable. Do not force unrelated work into one endlessly open trace.

Treat incoming trace context as untrusted correlation data, never as user or tenant authorization. Restrict baggage and custom fields so external callers cannot inject sensitive data, unlimited dimensions, or misleading operational labels.

Test a known request end to end. Confirm the expected parent or link relationships, service versions, error state, and log correlation. Test a delayed job and an uninstrumented dependency so the team recognizes a gap rather than interpreting a missing span as zero duration.

5. Validate the collection and sampling path

The telemetry platform owner defines collector deployment, transport security, queues, retry bounds, memory limits, and backend behavior. Pin supported component versions and test configuration changes before rolling them out.

Tail sampling can choose traces using observed outcomes, but it cannot keep every error or slow request by guarantee. It only evaluates spans that reach the relevant processor within its available capacity and decision behavior.

"type": "svg-architecture", "title": "Validate what reaches the sampling decision", "nodes": [ ], "links": [ ], "caption": "This is one tail-sampling pipeline. Loss can occur before or after the decision, so an error-retention policy is not proof that every error trace is stored." }} />

Review the deployed version of the OpenTelemetry tail-sampling processor. Its configuration includes decision timing, trace capacity, policies, and late-span behavior. Ensure spans for a trace reach the same sampling decision path, including during scaling and collector changes.

Inject known error and slow traces, late spans, collector restarts, and backend unavailability. Compare emitted test operations with retained evidence. Monitor queue pressure, refused or dropped data, exporter failures, and processing capacity.

Head sampling may discard a trace before an eventual error is known. Tail sampling adds state and resource demand. Choose the combination from investigation needs and measured cost, and report its limits. Do not compute an unbiased user error rate from outcome-biased sampled traces without a valid correction method.

For the sampling test, maintain an independent manifest of synthetic trace IDs and expected span relationships. Send cases across collector instances and during a planned instance replacement. Compare received spans, sampling decisions, exported spans, and backend query results. Do not count a trace as complete merely because its root span exists. Define which child spans are required to answer the operator's question and report partial traces separately.

Keep collector self-monitoring reachable when the destination backend is unhealthy. Otherwise the same outage can remove application evidence and the evidence explaining its loss. The telemetry owner chooses a bounded independent health path and tests its failure behavior. This does not require duplicating every signal into a second vendor; a small operational heartbeat or queue-health channel may answer the specific availability question.

6. Define an SLO and an error-budget policy

Choose the target from user needs, contractual obligations, current evidence, and feasible engineering changes. Setting it slightly above today's performance is not a complete business decision.

An additional nine reduces the allowed failure fraction tenfold for the same measure and window. For example, 99.9% permits 100 failures per 100,000 eligible operations, while 99.99% permits 10. This arithmetic does not imply ten times the engineering effort or cost.

Define how the team responds to budget consumption. Possible actions include a targeted reliability fix, restricted risky releases, or capacity work. Preserve emergency security and recovery paths rather than applying an indiscriminate release freeze.

Google's SRE workbook on alerting from SLOs explains burn-rate and multiwindow approaches. Use its reasoning to select and test local windows and thresholds. Low-traffic workloads, seasonal patterns, and delayed completion signals need particular care.

7. Connect each alert to a decision

The on-call owner decides whether a condition needs an immediate page, a scheduled ticket, a dashboard, or no retained signal. Record the risk, response deadline, owner, and evidence needed to act.

User symptoms are useful paging signals, but some leading risks deserve action before users are affected. Imminent data loss, expiring critical credentials, or exhaustion of a nonrecoverable resource may justify an alert under an explicit risk model.

An alert should identify the service and environment, explain impact, link to evidence and a runbook, and name the safe first action. Test routing, escalation, acknowledgment, and recovery notifications.

"type": "flow", "title": "Prove the operator path before enabling an alert", "steps": [ ], "caption": "An alert is ready when its detection and human response work together. Threshold correctness alone does not verify notification delivery or safe remediation." }} />

Group related symptoms without concealing independent failures. Define suppression scope and expiry. An alerting change must not permanently silence the only signal for a critical risk.

8. Review dormant alerts by risk coverage

An alert that has not fired may protect against a rare catastrophic event. Review its purpose, expected frequency, dependencies, testability, owner, and replacement coverage before retiring it.

Use synthetic or controlled tests to exercise dormant rules and delivery routes without causing the real incident. An unused runbook may need maintenance even when the alert remains necessary.

| Review finding | Decision | | --- | --- | | Rare but consequential risk remains | Retain and exercise detection and response | | Signal duplicates another verified control | Consolidate with an explicit coverage record | | Alert repeatedly lacks an action | Redesign the condition or move it to a suitable review channel | | Monitored component is retired | Confirm dependencies and remove the exact obsolete rule | | Telemetry or delivery is broken | Repair it; silence is not evidence of health |

Record the old rule, decision, approver, replacement, and recovery path. Keep versioned configuration so a mistaken retirement can be reversed.

9. Roll out telemetry without endangering the workload

Measure instrumentation overhead, export behavior, volume, and sensitive-field exposure before broad rollout. Bound buffers so a backend outage cannot consume unlimited application memory.

For ordinary diagnostic telemetry, define a degradation policy that protects the application while reporting lost visibility. Audit or security records may have different durability and fail-closed requirements; their owners must decide the behavior.

If a telemetry release causes excessive cost, latency, or disclosure risk, stop expansion and restore a tested configuration. Preserve necessary incident evidence under the approved handling policy. A sensitive-data incident may require access restriction, deletion procedures, credential rotation, and qualified review beyond reverting instrumentation.

10. Rehearse an investigation with incomplete evidence

The on-call owner runs an exercise using the export journey after instrumentation and alert routing are deployed. Introduce a safe test failure in the authorized environment, then ask another operator to identify affected jobs, distinguish a worker failure from collector loss, and choose the approved first action. Record what they used and where they had to ask the implementer for help.

| Exercise condition | Operator decision | Recovery evidence | | --- | --- | --- | | Queue age grows while workers remain healthy | Check dependency delay and admission before adding workers | Backlog drains without duplicate exports | | Application failures rise but error traces disappear | Check sampling and collection health | Known test errors become queryable again | | Backend is unavailable | Preserve bounded buffers or apply the approved loss policy | Export recovers without exhausting application memory | | Alert reaches the wrong team | Use the documented escalation path | Correct team receives and acknowledges a repeated test | | New instrumentation exposes a prohibited field | Contain access and involve the data owner | Field removal, destination assessment, and authorized cleanup evidence |

Rollback starts by identifying the last tested SDK, collector configuration, dashboard query, and alert rules. Reverting only one layer can leave renamed attributes or a changed SLI query incompatible with the others. The platform owner records the compatibility bundle and tests an export after the rollback. Data already dropped cannot be recovered by configuration alone; mark that investigation window as incomplete.

11. Sign off the investigation and response path

The service owner accepts the playbook when the SLI test matches the manifest, the sampling test describes measured loss, and another operator completes the investigation exercise. The on-call owner accepts alert routing and runbook actions. The data reviewer accepts field handling and destination access. Keep their decisions separate so a successful alert test is not mistaken for privacy approval.

Attach the telemetry contract, query revision, rollout comparison, failure manifest, response timeline, and restored configuration. Name the next review trigger: a queue replacement, schema change, new tenant-access model, collector upgrade, or altered completion deadline. For the next action, complete this evidence pack for one journey before instrumenting every service. DevOps and SRE covers the wider operating-system work when ownership or recovery is the unresolved constraint.

Reusable telemetry and alert record

Record the user question; operation and SLI definition; required events, metrics, and spans; field classification; cardinality budget; collection route; sampling policy and version; loss measurements; retention and access; alert condition; response owner; runbook; test evidence; rollback configuration; and next review trigger.

"The SLI has a defined numerator, denominator, window, and missing-data behavior.", "Telemetry fields, tenant access, retention, and cardinality have approved boundaries.", "Context propagation and business outcomes are verified with known test operations.", "Sampling tests include late spans, capacity pressure, restarts, and export failure.", "SLO targets do not imply a universal engineering-cost multiplier.", "Alert delivery and operator actions are tested, including low traffic and missing signals.", "Dormant alert retirement preserves explicit risk coverage and a reversible configuration.", "Instrumentation rollback and telemetry-loss behavior are documented." ]} />

Limitations

Telemetry provides selected evidence, not a complete recording of the system. Collection gaps, sampling, clock differences, and incorrect instrumentation can mislead investigations. Keep those limits visible and repeat the relevant tests when services, libraries, collectors, policies, or alert routes change.