Modern Data Lake Architecture: A Decision and Operating Framework

A practical whitepaper for deciding when a lakehouse is appropriate, defining data-product contracts, designing trusted data zones, assigning ownership, and proving...

audience="CTOs, heads of data, platform leaders, data architects and engineering teams deciding whether to modernize an analytical data platform." decision="Whether to keep, extend or replace the current warehouse or lake, and what technical and operating model should govern the chosen platform." position="A lakehouse succeeds when it is operated as a portfolio of owned data products with explicit contracts. Object storage and a table format are necessary components, not the operating model." scope="This paper is cloud and engine neutral. It covers analytical data platforms, not operational transaction stores, and uses illustrative workloads rather than promised performance or savings." outputs={[ 'A platform selection scorecard', 'A data-product contract template', 'Zone and promotion criteria', 'An ownership and control model', 'A recovery and observability checklist', 'A staged migration plan', ]} />

Abstract

Modern data platforms have more choices than the warehouse-versus-lake debate suggests. A team can retain a warehouse, add low-cost object storage, adopt an open table format, introduce streaming, separate storage from compute, publish domain-owned data products or combine several of these patterns. The architecture should follow the decisions the organization needs to make, the data those decisions require and the service levels that consumers expect.

This paper presents a decision framework for analytical platforms. It defines the evidence needed before choosing a target architecture, explains the role of raw, validated and published data zones, and specifies the controls that turn stored data into a dependable product. It also covers ingestion, table formats, data quality, lineage, access, workload isolation, recovery, operating ownership, cost and migration.

The central recommendation is straightforward: design the trust path before selecting the toolchain. A dataset becomes useful when a named owner can explain its meaning, freshness, quality, lineage, access policy and recovery behavior. A technically elegant lakehouse without those controls becomes another place to search for data that nobody is willing to defend.

Scope, assumptions and terms

This paper assumes an organization already has operational systems and at least one analytical workload. It does not assume a particular cloud, catalogue, table format or query engine. Examples use common terms such as bronze, silver and gold because readers recognize them, but the labels are not architecture. The promotion rules between the layers carry the design.

The terms below are used consistently:

| Term | Meaning in this paper | |---|---| | Data lake | Object storage that holds structured, semi-structured or unstructured analytical data. | | Lakehouse | A data lake combined with table metadata and transactional capabilities that support governed analytical workloads. | | Data product | A published dataset or interface with a named owner, contract, policy, service objective and consumers. | | Data contract | A versioned agreement covering structure, semantics, quality, delivery, classification and change. | | Promotion | A controlled change in trust state, such as preserved data becoming validated or validated data becoming published. | | Quarantine | A separately observable state for data that cannot safely advance. | | Replay | Reprocessing from preserved source evidence with a known code, contract and configuration version. | | Control plane | Catalogue, policy, orchestration, metadata, deployment and operational services used to manage the platform. |

The paper does not claim that a lakehouse always costs less or performs better than a warehouse. Those outcomes depend on data volume, concurrency, scan patterns, storage layout, engine behavior, staffing, licences, idle capacity and the cost of operating more components.

The decision before the architecture

Teams often begin with a product comparison. The more useful starting point is a decision inventory. List the business or operational decisions the platform must support, then trace the required datasets, freshness, history, controls and consumers.

For each important decision, record:

who makes it and how often;
which source events or records are required;
how late or incomplete the data may be before the decision changes;
whether the result must be reproducible later;
which identities may see row-level or sensitive fields;
how many concurrent users or automated consumers will query it;
how an incorrect result would be detected and corrected;
the maximum tolerable recovery time after data loss or corruption.

This inventory separates a platform requirement from a fashionable capability. A monthly finance close, a five-minute fraud signal, an ad hoc data-science exploration and an online recommendation lookup have different latency, correctness and serving needs. They may share governed data products without sharing the same compute engine.

Selection scorecard

Score the current and proposed platforms against observed workloads. Use a scale such as 1 to 5, but attach evidence to every score. A number without a query trace, incident, cost line or consumer requirement is only an opinion.

| Criterion | Evidence to collect | Warehouse tends to fit when | Lakehouse tends to fit when | Hybrid tends to fit when | |---|---|---|---|---| | Workload shape | Query history, data types, model-training reads | SQL reporting dominates | Diverse files, exploration and ML dominate | Reusable products serve several workload classes | | Concurrency | Peak sessions, queue time, workload classes | Predictable interactive demand | Large batch or elastic processing dominates | Interactive and batch require separate isolation | | Freshness | Source-to-decision latency | Scheduled loads meet the need | Streaming or incremental processing adds value | Some products need streaming while others do not | | Governance | Current catalogue, access and lineage gaps | Existing warehouse controls are effective | Open data access needs stronger cross-engine policy | Central policy must cover several serving systems | | Portability | Exit requirements and engine dependence | A managed platform is an accepted constraint | Open storage and table metadata are strategic | Storage portability matters more than every query path | | Recovery | Tested restore and replay time | Platform-native recovery meets objectives | Immutable source evidence and replay are required | Different products have different recovery paths | | Skills | Current on-call and engineering capability | SQL and warehouse operations are strong | Distributed processing skills already exist | A platform team can own the interfaces between systems | | Economics | Billed usage, idle capacity, licences, labour | Predictable load uses reserved capacity well | Variable workloads benefit from separated storage and compute | Serving and transformation can be costed independently |

Retaining the current platform is a valid outcome. Migration is justified only when a measured constraint, risk or strategic need exceeds the cost of change.

Worked scenario: a decision-led platform design

The scenario below is illustrative. It shows how to use the framework, not a client result or performance promise.

An industrial marketplace has four analytical needs:

Daily finance reporting from orders, payments and refunds.
Inventory alerts that should reflect source changes within ten minutes.
Supplier performance models trained on two years of order and logistics history.
Customer-facing order analytics with predictable sub-second dashboard interactions.

The source estate contains a PostgreSQL order system, a payment provider API, supplier CSV files and an event stream. Daily volume is moderate, but supplier files are inconsistent and late. Finance requires reproducible month-end numbers. Data science needs access to detailed history. Customer-facing analytics has higher concurrency than internal reporting.

A single engine is not the strongest design. The team chooses:

immutable source evidence in object storage, including source timestamps and replay positions;
an open table format for validated historical data;
incremental transforms for order and payment changes;
a published finance product with versioned business rules and close-period snapshots;
a low-latency serving store or warehouse for customer-facing dashboards;
a catalogue and policy layer that connects each published product to its owner, classification and consumers;
separate workload budgets for ingestion, transformation, model training and interactive serving.

The design is called a lakehouse because governed tables on object storage are central, but its value comes from the contracts and workload boundaries. Replacing the serving store with direct queries over every historical file would make the architecture simpler on a slide and less predictable for customers.

Scenario acceptance measures

| Concern | Proposed measure | Evidence before cutover | |---|---|---| | Finance reproducibility | Same source cutoff, rules version and table snapshot reproduce the approved close | Re-run two historical periods and reconcile every material variance | | Inventory freshness | 95th-percentile source-to-published delay within the agreed window | Measure by source, product and failure condition | | Supplier quality | Invalid records are quarantined with an owner and reason | Seed schema, duplicate and late-file failures | | Customer analytics | Representative dashboard concurrency meets its latency objective without delaying ingestion | Run mixed-workload tests, not isolated query benchmarks | | Recovery | A corrupted publication can be rebuilt from preserved evidence | Restore the catalogue state and replay one product in a clean environment | | Cost | Spend is attributable to ingestion, transformation, product and consumer workload | Reconcile cloud billing with engine and product usage for one month |

The reference architecture: a trust path

The architecture below follows a data product from its producer contract through preservation, validation and publication. This section defines what must be true at each boundary.

Producer boundary

A producer owns the meaning of its source. The platform team can automate extraction, but it cannot decide whether a null order date is valid, whether a status transition is legal or which timestamp represents the business event. The producer contract should contain:

a stable product and source identifier;
schema and field-level definitions;
event-time and ingestion-time semantics;
keys, uniqueness and ordering rules;
expected delivery pattern and lateness;
data classification and retention;
known corrections and deletion behavior;
compatibility and notice rules for change;
a named technical and business owner.

Contracts can be encoded in schema registries, table definitions, tests and catalogue metadata. The tool matters less than making the rules versioned, testable and visible to producers and consumers.

Preserved zone

The preserved or bronze zone answers, “What arrived, from where and when?” Keep source fidelity, ingestion metadata and a replay position. Avoid destructive cleansing. If files are malformed, store them separately with the failure reason rather than silently discarding them.

Preservation does not mean retaining everything forever. Retention follows legal, privacy, contractual and recovery needs. A platform may retain encrypted source evidence for 30 days, a year or longer, but the period should be explicit and tested against replay time.

Useful metadata includes:

| Field | Purpose | |---|---| | source_system | Connects the record to an accountable producer. | | source_object | Identifies the table, topic, endpoint or file. | | source_position | Supports deduplication and replay, such as an LSN, offset or object version. | | event_time | Represents when the business event occurred. | | ingested_at | Represents when the platform observed it. | | contract_version | Connects validation to a known rule set. | | ingestion_run_id | Connects rows to logs, code and orchestration state. | | content_hash | Supports duplicate and integrity checks where appropriate. |

Validated and conformed zone

The validated or silver zone contains data that has passed structural and domain checks. It normally includes a non-aggregated representation that preserves business detail. Common work includes type normalization, deduplication, late-arrival handling, reference-data resolution, privacy controls and historization.

Do not write directly from ingestion to a trusted silver table when a malformed or incompatible record could break the transform. The Azure Databricks medallion guidance recommends validating and cleaning data before it reaches the silver layer, including schema enforcement, deduplication and late-data handling.

Conformance should not erase disagreement. If two sources define “active customer” differently, record the definitions and choose a product-specific rule. Renaming both fields to the same column does not make the semantics consistent.

Published zone

The published or gold zone contains products designed for a consumer need. A product may be a dimensional model, an aggregate, a feature set, a secure view, an API or a governed file share. Publication requires more than a successful pipeline:

the owner accepts the semantics;
required quality checks pass;
access policy is active and tested;
lineage connects the product to inputs and code;
freshness and availability objectives are defined;
consumer compatibility is known;
cost and recovery behavior are observable;
documentation includes examples and limitations.

Gold should contain fewer, more purposeful products than silver. Copying every validated table into a “gold” schema only changes the folder name.

Dataset promotion and quarantine

Promotion should be a state transition with evidence. A workflow records the input snapshot, contract, code version, configuration, output identifiers, tests and publication decision. If a required check fails, the candidate remains outside the trusted product.

Quality dimensions and gates

“Data quality” is too broad to operate. Define dimensions against the consumer decision:

| Dimension | Example rule | Failure action | |---|---|---| | Completeness | Every settled payment has an order reference | Quarantine affected records and page the product owner when the threshold is material | | Validity | Currency is an allowed ISO code | Reject the record or map only through an approved reference table | | Uniqueness | One current order per source order identifier | Stop publication if duplicates would change totals | | Consistency | Refund total cannot exceed captured payment without an exception state | Isolate the exception and preserve the source history | | Timeliness | 95% of source changes arrive within the product freshness objective | Mark the product stale and notify consumers through a machine-readable status | | Referential integrity | Every line item resolves to an existing order version | Hold the affected partition or publication | | Distribution | Order volume and amount remain within an explainable range | Warn, compare with source and require review before publishing when impact is high |

Not every rule should block publication. Classify checks as blocking, warning or observational. The classification belongs in the contract and should reflect consumer impact. A missing optional marketing attribute may be a warning; an unreconciled finance total is a blocker.

Quarantine is an operating queue

A quarantine with no owner or service expectation becomes permanent storage. Every quarantined item needs:

the failed rule and observed value;
affected source, partition and product;
first-seen and last-retried timestamps;
accountable owner;
consumer impact;
correction path, such as source repair, rule change or accepted exception;
replay status and evidence.

Track quarantine age and volume by owner. A growing queue is a platform health signal, not an acceptable side effect of “keeping bad data out.”

Storage and table-format decisions

Columnar files such as Parquet are a common analytical storage choice because they allow engines to read selected columns and benefit from compression and statistics. The file format does not provide table-level transactions, snapshots, schema evolution or concurrent-write coordination by itself.

Open table formats add metadata and transaction rules above files. Delta Lake, Apache Iceberg and Apache Hudi overlap, but differ in engine support, metadata organization, streaming behavior and operational characteristics. Use current project documentation and a workload proof rather than a static feature checklist. Relevant primary references include the Apache Iceberg specification, Delta Lake documentation and Apache Hudi documentation.

Table-format evaluation worksheet

| Question | Evidence to produce | |---|---| | Which engines must read and write the table? | Tested compatibility matrix for exact versions and features used. | | How many files, partitions and snapshots will the table produce? | Metadata planning and commit tests at projected scale. | | How are concurrent writers coordinated? | Failure test with overlapping batch, stream and maintenance jobs. | | How are schema and partition changes applied? | Forward and rollback exercise with old and new readers. | | What does time travel guarantee? | Snapshot retention, deletion and privacy behavior. | | How are small files compacted? | Scheduled policy, workload isolation and before-after query evidence. | | How is corruption recovered? | Catalogue and metadata restoration plus object-version strategy. | | What creates lock-in? | Inventory of engine-specific SQL, APIs, catalogue features and operational tooling. |

“Open format” does not mean “zero switching cost.” SQL dialects, catalogue behavior, security policy, orchestration, observability and team knowledge can create stronger dependence than the bytes on storage.

Ingestion design

Choose ingestion by source behavior and decision latency.

Batch files and extracts

Batch remains appropriate when sources expose files, business processes operate on periods or a scheduled extract meets the consumer objective. A dependable batch design has a manifest, immutable object name or version, checksum, expected partition, record counts, idempotency key and late-file policy.

Do not use a successful file transfer as proof of a successful load. Reconcile source objects, bytes, rows and business totals. Retain enough evidence to distinguish “the source sent no records” from “the extraction failed before producing a file.”

Change data capture

CDC can reduce load on source databases and publish changes quickly, but it exposes source-specific behavior. Plan for snapshots, log positions, schema changes, transaction ordering, deletes, large transactions and connector downtime. The target must be idempotent because restarts can replay events.

A CDC pipeline needs a bootstrap protocol:

Capture a consistent source snapshot.
Record the log position associated with that snapshot.
Load and reconcile the snapshot.
Apply changes after the recorded position.
Verify no gap or duplicate changed the target state.
Preserve the ability to restart without guessing which side is authoritative.

Event streams

Use streams when the source already publishes meaningful events or when event-time processing changes the product outcome. Define ordering scope, watermark behavior, duplicate handling, late-data correction and replay retention.

“Exactly once” should be translated into observable behavior. Ask whether a business effect occurs once when producers retry, consumers restart, partitions rebalance and sinks partially fail. Transport guarantees do not automatically make downstream APIs, files or tables idempotent.

APIs and third-party SaaS

API ingestion must account for pagination, quotas, evolving fields, deleted records, partial responses and provider backfills. Store the request window, cursor, response identifier and extraction version. Reconcile against provider totals where available.

Transformation and semantic design

Transformation code should be versioned, tested and deployable through environments. Separate three concerns:

structural normalization, such as types, names and nested shapes;
domain logic, such as lifecycle, allocation and accounting rules;
consumer models, such as dimensions, aggregates and feature sets.

Mixing all three in one long query makes ownership and change impact unclear. A source schema change should not require a finance stakeholder to understand the entire ingestion implementation. A revised finance rule should not silently rewrite raw history.

Historization and correction

Define whether a product represents the latest known state, event history, effective history or a period snapshot. These are different products.

For data that arrives late or is corrected:

retain the source event time and observation time;
decide whether downstream results are corrected in place or restated as a new version;
identify affected products and periods through lineage;
notify consumers when a material published result changes;
keep enough history to explain the value visible at a prior decision time.

Governance and security

Governance should appear in the runtime path, not only in a policy document. The control plane connects identity, classification, policy, catalogue, lineage and audit.

Access model

Use stable groups and workload identities rather than direct grants to individuals. Separate platform administration, data ownership, stewardship, transformation execution and consumption. Privileged break-glass access should be time-bound, approved and audited.

Apply controls at the strongest practical layer. A dashboard filter is not row-level security. A catalogue label is not a storage permission. A masked view does not protect an unrestricted copy in a notebook workspace.

For each product, document:

allowed purposes and consumer groups;
row, column or object-level restrictions;
masking or tokenization rules;
export and downstream-sharing constraints;
retention and deletion behavior;
regional processing limits;
privileged access paths;
evidence retained for audit.

Privacy and deletion

Time travel, backups and copied analytical datasets complicate deletion. Define which layers retain identifiers, how a deletion request propagates, which snapshots remain under legal or recovery policy, and when physical purge occurs. Do not promise immediate deletion from every historical copy unless the architecture can prove it.

Lineage

Lineage must answer operational questions, not just draw a graph:

Which published numbers used a defective source partition?
Which consumers will break if this field changes?
Which code and contract version produced this table snapshot?
Which sensitive fields moved into an unapproved region or product?
Can this model result be reproduced from the same training data?

Capture lineage from orchestration, table metadata, transformation tools and serving interfaces. Validate it during incidents. An incomplete lineage graph that operators distrust is decoration.

Operating model and ownership

Platform team

The platform team provides reusable ingestion patterns, storage and catalogue integration, deployment pipelines, policy enforcement points, observability, workload controls and recovery tooling. It owns the paved road and its reliability, not the meaning of every dataset.

Domain producer

The producer owns source semantics, contract changes, domain quality and correction. It participates in incidents when the defect originates in the source or business rule. Ownership must include time and incentives, not only a name in the catalogue.

Governance and security

Governance defines classification, policy, retention and exception processes. Security validates identity, network, key management, privileged paths and control effectiveness. These teams should provide automatable rules and review paths rather than becoming manual approval queues for ordinary data work.

Consumer

Consumers state purpose, service need and compatibility expectations. They own efficient use, downstream controls and migration from deprecated versions. A consumer cannot demand “real time” without explaining which decision changes and what delay is acceptable.

Reliability and recovery

A data platform can be available while its products are wrong. Reliability covers control-plane availability, processing completion, product freshness, correctness, serving and recovery.

Failure domains

Review at least these failures:

| Failure | Containment question | Recovery evidence | |---|---|---| | Source unavailable | Does the product show stale status rather than current-looking old data? | Resume from a recorded position and reconcile the gap. | | Schema breaks | Can incompatible data remain outside trusted products? | Quarantine sample, alert owner and replay after correction. | | Partial transform | Can a consumer see half a publication? | Atomic commit or versioned publish pointer. | | Catalogue unavailable | Which existing reads continue and which writes pause? | Restore metadata and verify table snapshots. | | Bad business rule | Can lineage identify every affected product and period? | Rebuild with a corrected version and compare restated output. | | Region outage | Which products fail over, become read-only or remain unavailable? | Exercise the chosen behavior with measured recovery time. | | Credential compromise | Can access be revoked without breaking unrelated workloads? | Rotate, revoke and review audit evidence. | | Runaway query | Can one consumer exhaust shared capacity? | Enforce workload budgets and verify other classes remain within objectives. |

Recovery levels

Define recovery by product tier. A regulatory close product may need preserved source evidence, versioned business rules and period snapshots. An exploratory sandbox may accept re-creation from published products. Do not apply the same expensive recovery policy to every dataset.

Test recovery into a clean environment. Restoring objects without catalogue metadata, keys, permissions, table snapshots and orchestration state is not a recovered product.

Observability and service objectives

Monitor the user-visible data product and the machinery that creates it.

Product signals

freshness from business event to published availability;
completeness against source manifests or business totals;
failed and quarantined records;
contract and quality-test outcome;
active publication version;
consumer query errors and latency;
material corrections and restatements;
owner acknowledgement and recovery time.

Platform signals

ingestion lag, retries and source throttling;
orchestration queue time and failed dependencies;
compute saturation and workload-class contention;
object and metadata growth;
small-file and compaction backlog;
catalogue, policy and key-service availability;
cost by product, workload class and consumer;
restore and replay duration.

An SLO should name the product, window, measurement and exclusion. “Pipeline runs daily” is a schedule. “The finance-close product is published by 06:00 UTC on business days, with source reconciliation complete and no blocking quality failures” is an operational objective.

Performance and workload isolation

Start with representative queries and mixed workloads. Isolated benchmarks over a warm cache can hide contention, planning cost and metadata pressure.

Inspect:

bytes and files scanned per query;
pruning effectiveness and partition cardinality;
join shape, shuffle and spill;
concurrent interactive and background work;
cache hit behavior and cold-start time;
commit duration and metadata planning;
compaction and maintenance interference;
consumer cancellation and retry behavior.

Partition for common access and lifecycle patterns, not every filter. High-cardinality partitioning can create small files and excessive metadata. Clustering, sorting or engine-specific layout can improve pruning without turning each value into a partition.

Create workload classes for ingestion, transformation, maintenance, interactive BI, ad hoc exploration and model training. Give each class explicit concurrency, timeout and spend controls. A model-training scan should not delay the finance dashboard unless that trade-off is intentional.

Cost model

Cloud storage price is only one line. Build a product-level cost model:

monthly product cost =
  storage and retained snapshots
  + ingestion compute and transfer
  + transformation and maintenance compute
  + serving compute and cache
  + catalogue, observability and security services
  + engineering and on-call labour
  + migration and dual-run cost

Illustrative calculation

Suppose one product processes 20 TB of source data per month, retains 120 TB including history, runs 600 transformation compute-hours and serves 40 TB of queries. Do not apply a generic savings percentage. Price the exact storage tiers, engine rates, scan behavior, egress and reserved commitments. Then vary:

source growth by 2x and 5x;
query scan reduction from better layout;
idle versus elastic compute;
snapshot and replay retention;
maintenance workload;
dual-run period;
two realistic incident and recovery scenarios.

The model should reveal which variables drive cost. If serving scans dominate, moving storage will not solve the problem. If idle warehouse capacity dominates, elastic compute may help. If operational labour dominates, adding more engines may increase total cost despite cheaper infrastructure.

Migration strategy

Choose one decision path with visible pain and a willing owner. Avoid beginning with “move all data.” A thin slice includes the source, ingestion, contract, validation, publication, one representative consumer, observability and recovery.

Phase 1: baseline

Capture the current product’s data sources, semantics, query workload, freshness, incidents, cost and consumer dependencies. Preserve representative outputs and reconciliation totals.

Phase 2: contract and preservation

Define ownership and contract. Land immutable source evidence with replay metadata. Do not yet change the consumer.

Phase 3: build the trusted product

Implement validation, conformance, publication and policy. Run negative tests for unauthorized access and seeded quality failures.

Phase 4: compare paths

Run old and new outputs for the same source cutoff. Compare rows, aggregates, business meaning, latency, resource use and failure behavior. Differences need an owner and explanation.

Phase 5: move representative consumers

Migrate a consumer that exercises realistic joins, freshness and concurrency. Keep the old path available during the rollback window. Track user corrections and undocumented dependencies.

Phase 6: prove operations

Run restore, replay, source outage, schema break and runaway-query exercises. Verify on-call routing and product status communication.

Phase 7: retire deliberately

Remove the old path after reconciliation, consumer sign-off and rollback expiry. Archive required evidence, revoke credentials, stop duplicate pipelines and confirm cost actually leaves the bill.

Cases where a lakehouse is the wrong move

A lakehouse may not be the right next step when:

a well-operated warehouse already meets the workload and strategic needs;
the team lacks ownership and quality practices, and a new platform would only copy the disorder;
the workload is small, stable and entirely relational;
sub-second operational access is the primary need and analytical history is secondary;
the organization cannot operate the catalogue, policy, table maintenance and multiple engines it proposes;
data residency or product support requirements conflict with the chosen managed services;
the migration has no product owner, measurable constraint or retirement plan.

The right design can be a warehouse with better contracts and ownership, an object-storage archive beside the warehouse, or a smaller domain-specific lakehouse rather than an enterprise platform.

Architecture and operating review workbook

Use this section during design review. Record evidence and an owner for every “no” or “unknown.”

Purpose and consumers

[ ] The platform is tied to named decisions or products, not a general instruction to centralize data.
[ ] Each priority product has consumers, freshness and correctness expectations.
[ ] Current constraints are measured through workload, incident and cost evidence.
[ ] The target architecture explains why the current platform cannot reasonably meet the need.

Contracts and trust

[ ] Every priority source has a named producer and versioned contract.
[ ] Source evidence and replay positions are preserved for the required window.
[ ] Promotion rules distinguish blocking, warning and observational checks.
[ ] Quarantined data has an owner, reason, age and correction path.
[ ] Published products have definitions, limitations, examples and deprecation rules.

Security and governance

[ ] Identities and groups map to purpose and least privilege.
[ ] Sensitive fields are classified and controls are enforced in the data path.
[ ] Privileged access is time-bound and auditable.
[ ] Retention, deletion, snapshot and backup behavior are reconciled.
[ ] Lineage can answer incident and change-impact questions.

Reliability and operations

[ ] Product freshness, completeness and quality are visible to consumers.
[ ] Source, transform, catalogue, serving and regional failures have defined behavior.
[ ] Restore and replay have been tested in a clean environment.
[ ] Workload classes prevent one consumer from exhausting shared capacity.
[ ] On-call routes to the platform or product owner based on the failure.

Economics and migration

[ ] Cost includes storage, compute, serving, operations, labour and dual run.
[ ] Product and consumer usage can be attributed without building a billing system first.
[ ] One thin slice has explicit acceptance and rollback criteria.
[ ] Old and new outputs are reconciled for representative periods.
[ ] The old path has a retirement owner and date.

What to do Monday morning

Select one business decision that currently suffers from late, disputed or expensive data.
Write its source-to-consumer path on one page, including every copy and manual correction.
Name the producer and consumer owners.
Define five contract rules that would make the product defensible.
Capture current freshness, incident and cost evidence.
Decide whether the gap is architectural, operational or semantic before selecting a new platform.
Build the smallest thin slice that can prove preservation, validation, publication and recovery.

Primary references

These references describe platform patterns and current project behavior. Validate exact service limits, compatibility and recovery behavior against the versions and regions selected for an implementation.