Batch vs Stream Processing: When to Use Each

Choose batch, micro-batch, or streaming from decision deadlines, data correctness, replay requirements, and operating capacity, then test the recovery path.

trigger="A data product needs a new processing mode, or an existing pipeline misses a business deadline or cannot recover safely." owner="The data product owner accountable for the consumer's decision and correctness." participants={["Data engineer", "Source-system owner", "Consumer owner", "Platform operator", "Security and data governance owner"]} prerequisites={[ "A decision deadline, acceptable correction behavior, and representative source data.", "Known event identities, source retention, ordering guarantees, and schema ownership.", "Access to a representative test environment with controllable failures and isolated sinks." ]} outputs={[ "A processing-mode decision record and a costed operating model.", "A tested event-time, deduplication, checkpoint, and publication contract.", "A replay and reconciliation runbook with named stop and recovery authorities." ]} doneWhen={[ "Consumers receive sufficiently correct data before their actual decision deadline.", "Late, duplicate, missing, and out-of-order events produce the agreed result.", "A failure and representative backlog can be recovered within the approved recovery objective.", "Replay does not silently repeat payments, notifications, or other external actions." ]} />

Choose the decision contract before the engine

A dashboard that refreshes quickly but omits late sales may be useful for operations and unacceptable for settlement. A nightly dataset can be correct yet arrive after the decision it was meant to support. Neither pipeline speed nor a tool name resolves that tradeoff.

This guide chooses a processing mode for a data product. It is not an argument to turn every service interaction into an event. Use when async stops being worth it for that architectural boundary.

Batch processes a bounded input set. It can still maintain state, join historical data, and require careful recovery. Streaming continuously processes an unbounded input, often maintaining state across events. Micro-batch processes successive bounded increments and may be the execution mode of a streaming engine. These are not three fixed latency bands.

1. Record four clocks and the cost of being wrong

The consumer owner defines event time, arrival time, decision deadline, and correction window. Record timezone and timestamp quality as part of the contract. A source with unreliable clocks or delayed exports can dominate freshness regardless of processing speed.

| Clock | Question to settle | | --- | --- | | Event time | When did the business action occur, and who supplies that timestamp? | | Arrival time | When can the pipeline first observe it, including upstream delay? | | Decision deadline | When does a user or system actually need a usable result? | | Correction window | How long can prior results change, and who consumes corrections? |

Add the consequences of omission, duplication, stale values, and incorrect order. An approximate trend may tolerate revision. A payment or inventory commitment may require authoritative validation before action.

For an illustrative retailer, replenishment planners might accept an intraday provisional stock view, while finance closes sales after returns and delayed store uploads have been reconciled. The appropriate pipeline is not determined by the word “retail.” It depends on those consumers' separate contracts.

Gate: the consumer owner accepts the meaning of provisional and final data. If no one can state when a correction is allowed, postpone engine selection.

2. Compare feasible modes against the whole workload

The engineering lead tests candidates against volume, bursts, state, joins, source availability, and recovery. A nominal processing interval does not include queueing, startup, retries, publication, or upstream collection delays.

| Mode | Strong candidate when | Evidence that could rule it out | | --- | --- | --- | | Scheduled batch | A bounded dataset can finish before the deadline | Missed deadline during peak or replay; source export arrives too late | | Micro-batch | Incremental updates satisfy the consumer with bounded work per run | Overlapping runs, costly repeated scans, or backlog that never drains | | Continuous stream | Decisions require continuous updates and state can be operated safely | Unbounded state, absent replay source, inadequate on-call or sink guarantees | | Hybrid | Provisional and reconciled outputs have distinct accepted purposes | Two ungoverned definitions of truth or unaffordable duplicated operations |

Estimate sustained and burst throughput, skewed keys, state growth, checkpoint cost, storage retention, and downstream limits. Include engineering and on-call effort, not just compute. Batch is not necessarily cheaper; a large repeated scan can cost more than incremental processing. Streaming is not necessarily faster end to end if the source only delivers a daily export.

Choose the simplest option that meets the entire contract. A synchronous read or an incremental scheduled query may be sufficient. No universal sub-second cutoff justifies a streaming platform.

For the illustrative retailer, suppose planners approve a 15-minute freshness deadline and a test shows incremental micro-batches meet it under peak load. Select micro-batch for that provisional view, with a separate reconciled close for finance. If backlog recovery misses the deadline, the choice fails even if ordinary runs are fast. These are example requirements and observations, not recommended universal intervals or measured client results.

3. Design the result and publication boundary

Keep source capture, transformation, publication, and external actions separate. Preserve replayable inputs for the approved retention period, with access and deletion controls. Immutable for processing purposes does not mean exempt from data retention or erasure requirements.

The data engineer defines stable event IDs, business keys, ordering scope, schema compatibility, and output identity. An event ID is not necessarily a business-operation ID: a correction may be a distinct event about the same operation. Record how that correction replaces or adjusts the prior result.

For bounded jobs, use an input manifest or snapshot reference and an explicit completion marker. For streams, expose result freshness and revision state. Avoid presenting a partial batch or provisional aggregate as complete because the job process exited successfully.

4. Make lateness and duplicates visible

Event-time windows group events by when the business action occurred. Processing-time windows use the processing system's clock. Select intentionally; the same delayed upload can land in different windows under the two policies.

A watermark is an estimate of event-time completeness, not proof that no late event can arrive. Apache Beam's model documentation explains watermarks and triggers. Its programming guide describes allowed lateness and window behavior. Verify the selected runner's behavior rather than assuming a default retains every late event.

The consumer owner chooses whether late events update a published result, create a correction record, or enter a reviewed exception queue. State when a result becomes final and how rejected late data remains discoverable. Do not silently discard financially relevant events to make latency look better.

Deduplication needs a key and a retention horizon. If replay extends beyond retained deduplication state, duplicate protection may no longer hold. Test that boundary explicitly. For repeated window outputs, establish whether the sink receives replacements, accumulated totals, or deltas; treating a replacement total as a new delta double-counts usage.

5. Test checkpoint and sink guarantees together

A checkpoint restores a defined scope of processing state. It does not automatically reverse everything a process previously did. Apache Flink's checkpoint documentation describes recovery of managed state and corresponding stream positions, plus retention considerations. Select documentation for the deployed version and verify the actual source and sink connectors.

The engineer records which writes participate in a transaction or use idempotent identity. Test a crash after a sink write but before progress is recorded. Then test a recorded progress point whose downstream action is delayed or rejected. “Exactly once” is not an adequate acceptance statement without its boundaries.

For a reporting table, an idempotent replacement keyed by dataset version and business key may be appropriate. For email, payment, or fulfillment, persist a separately controlled intent and reconcile the external outcome. A retry can encounter an action that succeeded despite a timeout. Investigate the operation identity before repeating it.

Gate: the test report shows the final business result, not just a successful engine restart.

6. Rehearse backlog recovery and replay

The operator deliberately slows a downstream dependency and interrupts a worker in the test environment. Measure oldest unprocessed event age, processing rate, checkpoint duration, retained input, and sink error rate. More workers will not solve a hot key, database limit, or serial external API.

Set backpressure and admission policies from the consumer deadline and downstream safety limits. Do not let an unbounded queue conceal a failing service. Alert early enough that the remaining retained input and recovery capacity can still meet the approved objective.

1. The data owner authorizes the replay scope. Freeze the manifest or offset range, transformation revision, reference data and expected corrections.
2. The operator isolates replay writes. Use a candidate dataset and suppress external actions. Preserve the current consumer version.
3. The data owner reconciles the candidate. Compare keys, counts, totals, late records and known exceptions. Reject a run with unexplained differences.
4. The consumer owner approves publication. Record the version switch and recovery point. If acceptance fails, keep or restore the prior view and reconcile any decisions already taken.

A replay manifest records source ranges, schema versions, code revision, state or checkpoint origin, target namespace, expected counts, and excluded actions. Use a representative backlog, including skew and late corrections. Compare key-level differences and business totals against an independent reference.

For a small acceptance fixture, capture three distinct completed sales totaling 60 units, a duplicate of the first event, and a late correction reducing the third sale by 5 units. The reconciled result must be 55 units, with the correction traceable and the duplicate contributing nothing. Crash after writing a result but before recording progress, then replay the same manifest: the final result must still be 55, not 110. Repeat with the correction outside the configured late window and verify the agreed exception path, rather than silent loss. This fixture checks correctness; a separate representative-volume replay checks capacity and recovery time.

If a new result is wrong, stop publication and return consumers to the retained prior version where that is valid. That restores a dataset reference, not downstream decisions already made from bad data. The consumer owner must identify affected reports or actions and arrange corrections.

7. Allocate the freshness budget across the whole path

The data engineer turns the consumer deadline into a timing worksheet. Start at the business event, not at the point where the engine receives it. Include source publication, capture, scheduling or queueing, transformation, sink commit, and the consumer's refresh. Identify which owner controls each delay. Faster processing cannot recover time that the source has already consumed.

For the illustrative 15-minute planner deadline, suppose an event can spend eight minutes awaiting source publication, one minute in capture, three minutes waiting for and running an incremental job, and one minute reaching the visible result. The path consumes 13 minutes, leaving two minutes under those hypothetical assumptions. This is a planning budget, not a measured percentile or a claim that independent worst cases can be safely added for a statistical guarantee.

Test boundary cases with actual timestamps carried through the pipeline. Include an event just after a scheduling cutoff, a delayed source partition, and the busiest observed input mix. Record end-to-end age rather than adding unrelated percentile summaries. The consumer owner decides what the interface displays when the budget is exceeded: a stale-data marker, a withheld decision, or a permitted last-known result with its age.

The output is a budget with measured distributions, dependencies, and a named response to breach. If the source regularly delivers after the deadline, revisit the source contract or consumer requirement. Moving the same late export to a streaming engine changes neither constraint.

8. Define how changing reference data affects replay

Many pipelines join events to a customer category, product catalog, exchange rate, or another changing reference dataset. The data owner must specify whether a replay uses the value valid when the event occurred or the latest value at replay time. Both can be legitimate, but they answer different questions. An unchanged event stream can therefore produce a different result without an engine defect.

For a simple fixture, process a sale assigned to category A, change the product to category B, then replay the original sale. The expected category depends on the contract. If the report preserves historical classification, retain or reconstruct the applicable reference revision. If it intentionally restates history under today's classification, label the new dataset accordingly and record why it differs from the previous result.

The implementation owner records join keys, timestamp semantics, missing-reference behavior, and the retained versions needed for recovery. Test a reference update arriving before its related event, afterward, and outside the retained history. Route an unresolved historical lookup to an explicit exception path instead of silently substituting a convenient current value.

State grows with active keys, retained windows, deduplication history, and join requirements. Measure those quantities under a representative retention horizon and key distribution. A short prototype may not expose the eventual memory or checkpoint burden. The acceptance artifact must connect the chosen state policy to the correction and replay windows promised to the consumer.

9. Prove the write protocol for the selected engine

Use the engine's current connector contract as an implementation input. For example, Spark's Structured Streaming programming guide describes foreachBatch writes as at-least-once by default and explains using the batch identifier for deduplication. A custom callback that writes to two destinations still needs an end-to-end correctness design; the callback API does not make the two destinations one transaction.

The engineer chooses an output identity scoped to the dataset or query and the intended business effect. Do not assume a batch identifier is globally unique across independent queries or rebuilt checkpoints. For a replacement reporting dataset, write to a versioned target, verify completeness, and publish a pointer only after acceptance. For an external action, retain a durable operation identity and reconcile its effect separately.

| Injected failure | Expected observation | Acceptance evidence | | --- | --- | --- | | Worker stops before sink commit | No partial result is presented as final | Isolated target and completion marker | | Sink commits before checkpoint advances | Replayed output preserves the intended quantity | Stable output identity and final totals | | First destination succeeds and second fails | Inconsistency stays visible and owned | Per-destination status and repair record | | Required checkpoint or input expires | Recovery stops at the documented limit | Resnapshot or reconstruction decision | | Replay encounters erased source data | Exclusion follows the approved data policy | Manifest exception, not silent recreation |

Run the fixture against the deployed engine and connector versions. Capture the checkpoint origin, input range, output version, and failure point. The done-when condition is the agreed business result after restart, including any unresolved exception, rather than a green process status.

10. Rehearse publication rollback with the consumer

The operator and consumer owner prepare a candidate dataset and a retained prior version. Keep live actions disabled during the exercise. Introduce a known discrepancy, such as a duplicated aggregate or missing late correction, and verify that the publication gate rejects it before readers switch. Then rehearse discovering the discrepancy after a bounded test consumer has already read it.

Record what a publication rollback can restore: the previous dataset reference, view, or routing configuration. List what it cannot restore automatically, including exported spreadsheets, replenishment choices, notifications, or downstream writes already based on the faulty result. Those need a consumer-specific correction procedure and an accountable owner.

Test reader behavior during the switch. A consumer should not combine half of one dataset revision with half of another when the contract promises a coherent snapshot. Long-running exports may need to pin a version. Access rules and retention policies apply to prior versions as well as the latest result.

Close with a mode-selection record containing the measured deadline, tested input and state horizon, correctness fixture, recovery limit, and operating owner. The next action is the smallest representative experiment that could disprove the proposed mode. Keep a viable simpler alternative in the record so a failed stream or micro-batch trial can lead to a concrete decision rather than an open-ended platform expansion.

Reusable decision and recovery record

| Field | Required evidence | | --- | --- | | Consumer decision | Deadline, consequence of stale or wrong data, accountable owner | | Source contract | Identity, event time quality, schema owner, ordering scope, retention | | Mode selection | Candidates tested and specific reasons rejected | | Correctness | Window, late-event, duplicate, replacement, and finality rules | | Sink guarantee | Transaction or idempotency boundary and crash-test results | | Capacity | Peak, key skew, backlog volume, drain time, downstream limits | | Replay | Manifest, isolated target, comparison, promotion approval | | Recovery | Previous result, checkpoint availability, action reconciliation, escalation |

Acceptance checklist and limitations

"Decision and correction deadlines come from the consumer, not a generic latency rule.", "Representative late, duplicate, missing, and out-of-order events are in the test fixture.", "A failed run cannot expose an unmarked partial result as final.", "Checkpoint retention and source retention support the tested recovery scenario.", "Replay cannot send live payments or notifications without separate authorization.", "Backlog recovery meets the agreed objective without overwhelming the sink.", "An operator and consumer owner accept both publication and downstream correction procedures." ]} />

This framework does not certify any engine or connector's delivery guarantee. Versions, connector settings, external systems, and data contracts change what is achievable. Domain owners must validate financial, safety, and retention requirements. A prototype demonstrates feasibility only for the tested workload; retain its inputs and failure results so the decision can be revisited as volume or consumer needs change.