Did AWS DMS Validate the Whole Large Value?

Review AWS DMS LOB transfer and validation scope separately, expose matching-prefix blind spots, and record the endpoint, task and content evidence needed for a...

A passing AWS DMS validation result supports a full-value claim only when the actual endpoint, column, transfer configuration and comparison scope support that claim. A matching prefix cannot establish agreement for the remaining content. Equal row counts and equal value lengths cannot close that gap either.

Start with two separate questions: did the target receive every required part of the value, and did the comparison inspect every required part? A task can be configured to transfer more than its validation examines. A limited transfer and a limited comparison can also agree while both omit content the application needs.

This article gives a database migration reviewer a LOB coverage record and an offline counterexample. It focuses on AWS DMS task-based relational migration, not a generic cutover runbook or DMS homogeneous data migrations. All values and comparisons below are synthetic. No DMS task, endpoint or database was created or operated. The article makes no claim that any real migration passed or failed.

1. Pin the column and endpoint before interpreting a status

“LOB” is a DMS handling category, not a sufficient description of the source value. Record the source engine and exact version, source column type, DMS intermediate type, target engine and exact version, and intended target representation. Include the actual DMS engine version and task migration type. A table's transfer support does not establish its validation support.

For a documented example, PostgreSQL source BYTEA maps to DMS BLOB, while a PostgreSQL target maps DMS BLOB to BYTEA. PostgreSQL source OID LOBs are a different case: AWS documents that they are not migrated to the target. Do not apply a BYTEA conclusion to an OID reference. DMS PostgreSQL source types and limitations, DMS PostgreSQL target mappings

The planning example in the worksheet uses ordinary PostgreSQL 15.x source and target databases with a non-null integer primary key and a BYTEA payload. Exact deployed minor versions, endpoint settings and task engine readback are UNKNOWN because no deployment exists for this article. Treat those as missing real-world evidence, not a claim that every 15.x combination has been tested. Babelfish, Aurora PostgreSQL Limitless, Oracle-specific handling, character LOB conversions and S3 targets require their own support review.

Obtain the effective task settings, table mappings and relevant endpoint settings from authorized read-only evidence. Preserve their version or hash with the result. A screenshot of a console label without the configuration that gave it meaning is insufficient for a later reviewer. Do not change a production task simply to discover what its current setting does.

2. Read transfer scope and validation scope independently

The general DMS LOB guide distinguishes full, limited and inline handling. Full mode transfers LOBs in pieces. Limited mode accepts a configured maximum and documents truncation with a warning for larger values. During full load, inline LOB handling transfers smaller values inline and uses full mode for larger ones; it is unavailable where full LOB mode is unsupported. A full-load observation does not establish subsequent CDC handling. Retain the actual phase, settings and endpoint support in the evidence record. For limited LOB validation, AWS instructs that ValidationPartialLobSize match LobMaxSize. DMS LOB support

Record endpoint-specific behavior too. For a PostgreSQL source, FailTasksOnLobTruncation=true causes a limited-LOB task to fail on an oversized value instead of truncating it; the documented default is false. Do not write a universal “DMS always truncates” incident explanation without checking that setting. Nor does enabling that guard prove previously transferred content is complete. PostgreSQL source endpoint settings

The corresponding transfer parameters live in target metadata, including SupportLobs, FullLobMode, LimitedSizeLobMode, LobMaxSize, LobChunkSize and InlineLobMaxSize. Table-specific management can also apply. The InlineLobMaxSize description explicitly applies to full load. A chunk size describes transfer pieces; it is not the number of bytes validated. DMS target metadata settings

For comparison, EnableValidation enables task validation. SkipLobColumns=true excludes LOB columns. ValidationPartialLobSize defines partial comparison in KB-labelled units; zero means all LOB column data, within the service's applicable support and exclusions. Setting zero cannot undo a skipped column. DMS validation settings

Treat those settings as evidence to interpret, not a paste-ready migration recipe. The correct combination depends on the endpoint pair, actual values and accepted content contract. Keep source units as documented. Do not assume a text character count, encoded byte length and a KB-labelled service parameter are interchangeable.

3. Decide whether the required content fits the proposed mode

Write the requirement before reviewing a passing result. A binary attachment may need exact complete bytes. A deliberately extracted preview may require only a specified leading segment, with the full attachment retained elsewhere under a separate contract. Those are different deliverables. An accidentally truncated attachment cannot become an approved preview after a prefix check passes.

Full-content transfer and full supported comparison

What the reviewer must establish: Entire required values were observed and compared under the actual endpoint rules

What remains unproven by a prefix agreement: Business meaning, untested rows and later changes

Full-content transfer and partial comparison

What the reviewer must establish: Transfer completeness separately; compared range and omitted range explicitly

What remains unproven by a prefix agreement: Agreement of the unexamined suffix

Limited transfer with matching partial validation

What the reviewer must establish: Intended limit, observed size population, oversized handling and whether truncation is acceptable

What remains unproven by a prefix agreement: Preservation of source content beyond the limit

LOB validation skipped

What the reviewer must establish: Exclusion is explicit and another accepted evidence method covers the requirement

What remains unproven by a prefix agreement: Any LOB content equality from that DMS result

Unknown effective settings or unsupported column path

What the reviewer must establish: Obtain exact support/configuration evidence before interpreting the result

What remains unproven by a prefix agreement: A defensible full-value acceptance claim

For a complete-content requirement, a known truncation is a failure even if the permitted prefix matches. If limited mode is proposed because the known population fits its limit, retain the size evidence and its observation boundary. A new larger value can invalidate that argument. Determine how new writes and an oversized value are detected and handled rather than relying on a historical maximum indefinitely.

Where the proposed native comparison cannot cover the required contents, evaluate another supported transfer/configuration arrangement or an independently designed content check. Do not disable a warning, skip the difficult column or narrow the requirement merely to obtain a passing status. Any deliberate scope reduction needs the data owner's approval and a clear account of what remains available to the application.

4. Run the equal-prefix counterexample locally

The offline fixture generates two complete binary observations of 128,000 bytes each. Every source byte is hexadecimal 41. The target is identical except that byte offset 96,000, using zero-based indexing, is hexadecimal 42. Both observations therefore have the same length and the same first 64,000 bytes. They represent one corresponding row each, so row counts match too.

The local comparator inspects the first 64,000 bytes and reports PREFIX_AGREEMENT_ONLY. It leaves 64,000 bytes per value unexamined: 128,000 minus 64,000. That is half of each value. A complete byte comparison reports CONTENT_MISMATCH because it reaches the known mutation. The expected failure comes from the fixture construction, not from whichever comparator result looks convenient.

These are explicit byte counts chosen for an inspectable local example. They are not DMS task parameter values, a conversion from AWS KB units, or an emulation of DMS SQL, hashing or validation states. The fixture demonstrates the logical coverage gap; it predicts no real task result.

Now replace the target with only the first 64,000 source bytes. The same local prefix comparator still agrees. The transfer scope has changed: the target is shorter and the source suffix is absent. Complete comparison fails on that difference. Two prefix agreements can therefore conceal different problems: an unexamined corrupted suffix, or a missing suffix. Retain observed lengths as well as range results, while remembering that equal lengths alone cannot distinguish the first case.

*Blue segments are the 64,000 locally inspected bytes. Hatched segments contain observed but unexamined bytes; the dashed empty segment is absent target content. The orange marker identifies the stipulated mutation, not an AWS event. Aligned strip lengths show the two local observation boundaries, not a measured transfer path. Both full local comparisons fail for different reasons. No DMS task or transfer was executed.*

5. Keep native outcomes separate from local coverage labels

AWS DMS reports validated, pending, mismatched and suspended states, among others. Continuous changes can prevent comparison, and validation adds database and network work. Native support requires an appropriate primary key or unique index, with additional key/type restrictions. Columns with data-masking transformations are ignored; a masking rule on a primary or unique key can cause the table to be skipped. DMS validation behavior and limitations

Record the actual provider result verbatim alongside the coverage interpretation. Do not rename a suspended record “failed content,” or treat no reported failures as evidence that a skipped LOB was compared. A table-level validated result must still be read within the effective column and comparison scope.

The following labels belong to this article's local review, not the AWS API:

Full complete observations agree at a defensible paired revision

Local interpretation: Full local agreement for those observations

Next review action: Preserve scope and inspect native/remaining acceptance evidence separately

Prefix agrees while required suffix is unexamined

Local interpretation: Partial coverage, not full agreement

Next review action: Obtain supported complete-content evidence

Compared contents differ

Local interpretation: Content mismatch in the observed pair

Next review action: Investigate identity, revision, extraction and transfer before authorized repair

LOB excluded from the comparison

Local interpretation: Excluded

Next review action: Retain the exclusion and a separate evidence requirement

Observation missing, incomplete or revision not paired

Local interpretation: Unresolved

Next review action: Establish a complete comparable observation before drawing a content conclusion

The broader modernization reconciliation whitepaper owns paired data cuts, business invariants and authority transfer. Here, preserve those prerequisites while adding the exact LOB transfer and inspected-range record. A local byte agreement does not authorize cutover or establish that the replacement application interprets the value correctly.

6. Fill a coverage record for each required LOB column

Use stable table, row and column identities. Record data without putting sensitive payloads into a ticket. Controlled evidence references can identify the approved observations; raw documents and hashes may themselves require restricted handling. Access to comparison data does not authorize updates, reloading tables or repairing a live target.

Requirement

Filled synthetic/planning example: Exact complete binary payload, no approved truncation

Evidence still required for a real task: Data-owner approval of the content contract

Endpoint/type path

Filled synthetic/planning example: Proposed PostgreSQL 15.x `BYTEA` → DMS `BLOB` → PostgreSQL 15.x `BYTEA`

Evidence still required for a real task: Exact deployed minors, DMS engine, endpoint settings and current support

Task identity, engine and phase

Filled synthetic/planning example: No task exists; engine, migration type and phase UNKNOWN

Evidence still required for a real task: Actual task reference, effective configuration revision, full-load/CDC scope and observation time

Identity and observation boundary

Filled synthetic/planning example: One fictional integer-keyed row; paired revision stipulated

Evidence still required for a real task: Actual stable keys and defensible complete observations

Transfer mode and limits

Filled synthetic/planning example: No DMS transfer executed; full observations supplied locally

Evidence still required for a real task: Effective task/table LOB settings and endpoint oversized behavior

Validation settings

Filled synthetic/planning example: No native validation run; local prefix 64,000 bytes

Evidence still required for a real task: `EnableValidation`, `SkipLobColumns`, partial limit, mappings and effective exclusions

Observed content sizes

Filled synthetic/planning example: Source 128,000 bytes; target 128,000 bytes

Evidence still required for a real task: Complete source and target extraction evidence with units

Local inspected and omitted ranges

Filled synthetic/planning example: First 64,000 bytes inspected; remaining 64,000 per value omitted

Evidence still required for a real task: Provider-supported actual inspected scope, not an inferred range from the local fixture

Results and disposition

Filled synthetic/planning example: Local prefix agreement, complete mismatch; HOLD complete-content acceptance

Evidence still required for a real task: Native result, discrepancy owner, resolution evidence and authorized review

For an empty record, copy these field labels and enter the actual evidence reference, observed value, UNKNOWN where necessary, owner and required next action for each. Add the task/configuration version and review date. A result without the settings revision is not reusable evidence after settings change.

If source and target observations are taken at different revisions, leave the content decision unresolved even when the lengths match. The dual-run comparison article explains this pairing boundary. Do not use a broad content checksum to hide an unknown row identity or a moving observation.

7. Test the comparator with failures it must detect

The offline source package contains only coverage.mjs, coverage.test.mjs, scenario.json and README.md. Extract all four files into one directory, then run the tests there with Node.js 22 or later. No packages, AWS access or database are required.

node --test coverage.test.mjs

The tests intentionally let the prefix comparator miss the suffix mutation, then require the complete comparator to detect it. They also cover a truncated target, a changed first byte, the last included byte, the first excluded byte, the last value byte, missing observations, unpaired revisions and explicit exclusions. A deterministic sweep places suffix mutations across 2,238 small cases; none may be promoted to complete equality by the prefix result.

An empty binary value is distinct from a missing observation or null. The fixture supplies no database null-comparison policy, so null remains unresolved rather than being normalized to an empty buffer. It rejects invalid prefix bounds and unknown scope labels. These checks test the local code's contract, not DMS behavior.

Before relying on a real extraction or comparator, establish that it reads the entire required value. A driver, export tool or diagnostic view that returns only a preview can make two extracted buffers agree while both omit the original suffix. A complete comparison of incomplete observations is still incomplete evidence. Retain extraction method, length interpretation, read permissions, errors and pairing evidence.

If hashing is selected for a separate content check, define the complete input representation and supported extraction method, retain length and identity, and evaluate its cryptographic and operational limits. This article does not supply a database hash-query workaround or assume the same text encoding at both endpoints. Binary byte equality and semantic text equivalence require different comparison contracts.

8. Close the coverage gap before changing the conclusion

For a native-supported full-value check, confirm the exact endpoint/type path, effective LOB settings, non-skipped comparison scope, supported keys and complete native outcome. Keep masking and other exclusions visible. For a limited-mode design, follow the documented matching partial-validation requirement and determine separately whether omitting oversized source content violates the workload contract.

If native coverage cannot establish the required agreement, use an approved complete-content evidence method or revise the migration design. Schedule validation within the actual source/target capacity and data-handling limits. A read-only check can still create load or expose sensitive content. Stop and preserve incomplete coverage when the observation method fails; do not classify that interruption as agreement.

After a correction, collect a fresh comparable pair and retain the old failure with its resolution. A changed transfer mode, column mapping, endpoint version, new oversized value or revised comparison limit can invalidate prior evidence. This article deliberately stops before task modification, reload, repair or cutover commands. Those actions need the migration's separate authorization and recovery procedure.

Take one required LOB column through the coverage worksheet. If the team cannot show both where its complete required content went and which content the accepted comparison inspected, keep that column's full-value acceptance on HOLD and assign the missing evidence to an owner.