A Scheduled Demand Forecasting Pipeline on AWS

Ampity deployed a forecasting pipeline that sequences data export, preparation, model execution and output formatting. EventBridge starts the workflow, Step Functions...

[Problem] A forecasting model is only useful if its inputs arrive

The customer needed a repeatable way to run demand forecasts from source data. A successful model execution alone was not enough. Data had to be exported, prepared, passed to the forecasting job and formatted into usable outputs. Each stage depended on the previous stage completing with the expected files and structure.

Ampity deployed a new AWS pipeline for this driver-based forecasting workload. This case covers that pipeline only. The customer's application runtime, partner file exchange and separate development environment were other workstreams, not additional components of this workflow. The customer is unnamed to preserve confidentiality.

[Decision] Separate orchestration from model compute

The forecasting workload required more memory than the lightweight preparation jobs. Rather than force every step onto the same compute service, the delivered design separated their responsibilities.

An EventBridge schedule starts a Step Functions state machine weekly. The state machine sequences the jobs and catches failures. An ECS task on Fargate handles export, AWS Glue handles preprocessing and postprocessing, and AWS Batch on Fargate runs the forecasting model. Each stage reads and writes the S3 data lake.

This is the delivered job sequence recorded in the customer story. It is not a claim that every forecasting workload needs these services, or that changing a model provider would be a drop-in replacement.

[Flow] From export to formatted output

The figure reconstructs the documented workflow, not the account or network layout. Solid arrows show execution order. Dashed arrows show storage reads and writes. The table below gives the same sequence in text.

The pipeline uses fixed S3 folder conventions for extraction, staging, output and data. Those handoffs matter because a job can finish without producing the files the next job expects. Ampity's work included aligning paths and column names between scripts, not simply connecting service icons.

[Validation] Test the data contracts, not only the jobs

Full runs were exercised in QA more than once. Missing input files, mismatched column names and inconsistent paths were identified and corrected by aligning the contracts between stages.

The same state-machine design was deployed in QA and production. The customer story records a production execution that flagged expected inputs had not yet been uploaded. That observation matters: the workflow exposed a missing prerequisite rather than silently implying that a forecast had completed.

Step Functions handles workflow failures, CloudWatch Logs provides job-level visibility, and SES reports the export step's success or failure. The email scope is deliberately specific. This case does not claim that every downstream failure generated an email or that a complete incident-management process was demonstrated.

[Result] A scheduled workflow with visible failure boundaries

The delivered result was a pipeline from data export through formatted forecast output, with a scheduled start, defined storage handoffs and visible failure reporting. The source records full executions, but it does not establish a forecasting-accuracy improvement, commercial return or universal execution-time guarantee. None is claimed here.

For a similar workload, the first design question is where the process can stop without somebody noticing. Review input arrival, schema contracts, job sequencing and result availability before choosing additional infrastructure. That recommendation is guidance for another engagement, not an extra outcome attributed to this customer.

[Review] Questions before adopting this pattern

For another forecasting pipeline, agree how each stage proves completion. A successful compute job, an output file and a usable business forecast are different observations. Define the expected input version, output location and responsible reviewer before treating a workflow status as business acceptance.

Check the failure path with the same care as the successful run. A missing file should be distinguishable from a malformed file, a denied read and a compute failure. The delivered case demonstrates missing-input detection; it does not establish that every possible failure was exercised. A new implementation should record its own test results and notification coverage.

AWS documents Step Functions error handling and its limits. Retry and Catch behavior depends on the state and error type. A workflow author must inspect those rules rather than assume a catch-all handler covers every execution failure. This reference explains the service behavior, not the contents of the customer's state-machine definition.

Also agree who may restart an interrupted execution and how the team avoids accepting old output as a new forecast. The current story does not document duplicate-run protection or output versioning, so these remain review questions for a future design rather than claimed features of the delivered system.

Explore AWS consulting and migration, or discuss a data pipeline with Ampity.