CI/CD Pipeline Optimization

Reduce delivery wait time with measured pipeline experiments, trustworthy caches, explicit test coverage, immutable artifacts, and rehearsed release recovery.

trigger="Delivery feedback is slow or unreliable, and the team can identify a pipeline or release path to improve." owner="The platform or delivery engineering lead accountable for the selected pipeline." participants={["Service owner", "Test owner", "Security reviewer", "Release operator", "Developer representative"]} prerequisites={[ "Run-level timing and failure data, including queued, canceled, retried, and unsuccessful runs.", "A versioned pipeline, representative change examples, and a clean-build path.", "Documented release checks, artifact retention needs, and application and data recovery procedures." ]} outputs={[ "A critical-path baseline and a ranked optimization experiment record.", "A cache and test-selection contract with correctness and trust-boundary tests.", "A release manifest, recovery decision record, and comparison of feedback time, cost, and failure behavior." ]} doneWhen={[ "Representative changes receive faster useful feedback without unapproved coverage or security reductions.", "Clean and cached runs pass the same acceptance checks, including dependency and toolchain changes.", "Operators can identify the deployed artifact and safely halt a release.", "The service owner accepts tested rollback or forward-recovery evidence for application, data, and external effects." ]} />

Optimize useful feedback, not a headline duration

A pipeline is useful when it tells a developer whether a change is safe enough to advance and gives an operator a traceable release. A shorter green run is not an improvement if it skipped the only check that would have caught a defect.

Start with the team's actual feedback loop. Measure queue time, execution, manual waiting, retries, and the time needed to understand a failure. Compare similar changes and include failed runs. There is no universal duration after which every developer loses focus, and no standard percentage of pipeline work that caching will remove.

This playbook covers one existing delivery path. It does not require replacing the CI provider, rewriting the build system, or adopting every technique below. Select the bottleneck supported by evidence, then change one mechanism at a time.

1. Build a baseline the service owner can trust

The delivery lead collects run identifiers, commit and branch, change type, runner class, dependency state, queue timestamps, job start and end times, test results, retries, artifact identifiers, and the final disposition. Separate pull-request validation from release work and scheduled maintenance.

Report the distribution of feedback time and the slowest recurring paths. Averages can hide a queue that stalls only during busy periods. Compare cold and warm caches separately. Include runner charges, storage, transfer, and engineering effort when estimating operating cost.

Ask developers which failure messages are actionable and which require manual investigation. Record flaky tests as a separate category rather than treating every eventual green retry as a successful first attempt.

| Observation | Candidate investigation | Evidence before changing | | --- | --- | --- | | Long queue, short execution | Capacity, concurrency limits, job priorities | Arrival pattern, runner availability, and cost | | Slow dependency preparation | Package download cache or image preparation | Transfer, verification, install, and script timings | | One long test shard | Test distribution or expensive shared setup | Per-test timing and resource contention | | Repeated failed releases | Environment, contract, or migration gaps | Incident and release evidence | | Long manual approval wait | Approval scope and reviewer availability | Why the gate exists and who may change it |

Gate: the service owner agrees which metric improves and which guardrails must not worsen. Set locally justified limits for escaped defects, security findings, release failures, and cost. Do not convert an approval requirement into an automatic pass to improve a timing chart.

2. Trace the critical path and preserve dependencies

The pipeline owner draws which jobs depend on which outputs. Jobs can run concurrently only when their inputs, test data, environment, and resource use permit it. Parallel jobs that fight over one database may produce slower and less reliable feedback.

Put inexpensive, informative checks early when they do not suppress required later evidence. Cancel superseded pull-request runs where safe, but do not cancel a partially completed deployment or data migration as if it were a disposable lint job.

If build and test run independently, prove that the tested source, dependencies, configuration assumptions, and released artifact match. An immutable artifact digest is more useful for promotion than a mutable tag such as “latest.”

3. Treat a cache as an optimization with a clean fallback

An unchanged lockfile alone is not enough to skip installation. Native dependencies, operating system, architecture, runtime, package-manager version, install options, lifecycle scripts, generated files, and environment-dependent build inputs can change the result.

Prefer the package manager's supported download cache before caching a ready-to-execute dependency directory. With npm, a cache of downloaded packages can accompany a normal clean installation. The npm ci documentation explains lockfile consistency, replacement of an existing dependency tree, and the need to preserve relevant install flags. A cache hit is not evidence that installation or integrity checks can be omitted.

Define a compatibility key for the cached object. A compiled artifact needs different inputs than a download archive. Broad fallback keys may be acceptable for reusable downloads but unsafe for generated binaries. Include a cache-format version so the team can invalidate a broken design.

The GitHub dependency caching reference distinguishes exact and partial matches and documents cache access boundaries. Keep secrets outside cached paths and prevent lower-trust jobs from supplying executable cache content to privileged workflows.

The test owner exercises cold start, exact hit, partial restore, missing cache, corrupted content, dependency changes, runtime changes, and architecture changes. On a correctness mismatch, bypass the cache, rebuild from declared inputs, and quarantine the suspect entry. Cache unavailability should make the pipeline slower, not silently less complete.

4. Reduce test work only with an explicit coverage contract

For a monorepo, affected-project selection requires a maintained dependency graph, including shared configuration, generated code, fixtures, build tooling, and transitive dependencies. A file-path filter alone is not proof that other projects are unaffected.

The test owner defines which change classes require full validation. Unknown dependencies, graph failures, and shared build changes should fall back to that full path. Compare selected and full-suite results during a trial, then keep an agreed periodic or release-level full check.

Shard tests using observed duration and independent fixtures. Isolate database namespaces, network ports, credentials, and mutable test accounts. Increasing runner count will not fix a rate-limited external dependency.

Give quarantined tests an owner, issue, expiry, and replacement coverage or explicit risk acceptance. Preserve the first failure when a retry is used to diagnose nondeterminism. Do not allow an unlimited retry policy to convert intermittent defects into green releases.

Gate: a faster run is accepted only if the test owner can explain what it covers, what it omits, and how omissions are controlled.

5. Protect the delivery trust boundary

Separate untrusted contribution validation from publishing and deployment authority. Review workflow triggers, token scopes, runner persistence, secret exposure, and third-party actions. A self-hosted runner that executes untrusted code needs an isolation and cleanup design appropriate to that threat.

Use short-lived, scoped deployment credentials where supported. When using identity federation, restrict the trust policy to the intended repository, workflow, branch or environment, and audience. Removing a stored cloud key does not make an overly broad trust policy safe.

GitHub's secure-use guidance describes workflow hardening, least-privilege tokens, untrusted input, and action pinning. Review these controls alongside speed changes. A new cache or runner pool changes the attack surface.

Retain enough artifact, provenance, approval, and test evidence for investigation and recovery. A storage cleanup policy must account for deployed releases and required retention. When a secret is found, invoke the incident process for containment and replacement; deleting the log or commit is not credential revocation.

6. Choose a recovery path before deployment

The release owner records application, configuration, schema, data, queued work, and external effects separately. Redeploying an old binary cannot undo a deleted column, a sent email, or a completed payment.

| Change | Recovery condition | Required rehearsal | | --- | --- | --- | | Application-only change | Previous artifact still works with current contracts and state | Route traffic back and verify business behavior | | Additive schema change | Old and new application versions tolerate the expanded schema | Mixed-version read/write tests | | Destructive migration | Data recovery or forward repair is explicitly designed | Restore or repair in an isolated environment | | External side effect | A compensating action exists and is authorized | Reconcile operation IDs and avoid duplicate actions |

Use expand-and-contract changes when compatibility permits: introduce the new shape, migrate safely, verify both versions, and remove the old shape only after its rollback window closes. Some changes need a forward fix or restore rather than rollback. Record recovery-time and data-loss objectives from the service's requirements, not a universal five-minute promise.

7. Run an optimization experiment with a stop condition

1. The service owner accepts the baseline. Fix the change class, cache state, required checks and comparison metrics before the experiment.
2. The delivery engineer runs one bounded trial. Change a cache, dependency selection rule or parallel stage while keeping the release checks intact.
3. The team compares results and guardrails. Include failed, cancelled and cold-cache runs, runner cost, missing-check risk and recovery behavior.
4. The service owner keeps, narrows or reverses the change. Record the evidence and restore the prior pipeline configuration if the guardrails fail. Review already-produced artifacts separately.

Use this record for each experiment:

Pipeline and owner:
Hypothesis and measured bottleneck:
Representative change set and baseline period:
Configuration revision and candidate revision:
Primary metric, sample count, and comparison method:
Coverage, security, reliability, and cost guardrails:
Cache compatibility or test-selection assumptions:
Stop signal and person authorized to stop:
Bypass or configuration-revert procedure:
Observed result, limitations, and decision:
Evidence links and follow-up owner:

Start with a bounded repository, job, or environment. Observe enough representative changes to include the conditions that produced the original bottleneck. Explain differing workload mix rather than claiming all improvement came from the experiment.

8. Work one critical-path example before buying capacity

The following timings are an illustrative fixture, not an Ampity performance result. Suppose a change waits eight minutes for a runner, spends four minutes preparing dependencies, then runs three independent checks taking two, six, and nine minutes. Packaging takes another three minutes after all checks complete. With sufficient isolated workers, useful artifact readiness is eight plus four plus nine plus three: 24 minutes. Summing all three check durations would incorrectly describe elapsed time as 32 minutes.

This fixture gives the engineer several different hypotheses. Reducing the two-minute check to one minute changes no artifact-ready time while the nine-minute check remains the bottleneck. It may still improve early feedback for the defects that check catches, so record that as a different outcome. Removing four minutes of queueing can help without changing any test. Splitting the longest check might help, but only if setup costs and contention do not erase the gain.

The pipeline owner annotates the job graph with actual start and finish times, required outputs, resource constraints, and the first actionable failure. Run-level evidence should distinguish developer feedback from release readiness. A failure found early can save waiting even when successful runs take the same time. A job canceled because a newer commit superseded it should not be treated as a successful fast run.

Choose one hypothesis for the next experiment and state why it lies on the relevant path. Keep the existing implementation available as the comparison and recovery path. The artifact is a timing graph plus a question that can be disproved, not a general instruction to increase parallelism.

9. Make the cache experiment reproducible

For an npm-based example, start with downloaded package data and continue to run the declared clean-install command. The official setup-node documentation distinguishes package-manager caching from caching node_modules. Check the selected action revision and configuration; do not assume its current defaults match an older workflow copied into the repository.

The engineer writes a cache contract identifying the cached object, permitted producers and consumers, key inputs, restoration policy, and clean fallback. A download cache can tolerate different reuse than a compiled executable. If the team later proposes caching generated binaries, review the full build-input and trust model rather than inheriting the download cache's broad restore key.

| Test variation | Expected behavior | Evidence | | --- | --- | --- | | Cache absent | Clean install and required checks still complete | Cold-run timing and results | | Exact compatible hit | Preparation may improve; checks remain unchanged | Key, restored object, and check set | | Lockfile or install flags change | Dependency tree follows the new declared inputs | Installation and dependency comparison | | Runtime, architecture, or OS changes | Incompatible generated content is not reused | Key decision and native-module tests | | Corrupt or suspicious cache | Entry is rejected or bypassed, then rebuilt safely | Clean fallback and incident triage if needed | | Lower-trust job offers executable content | Privileged workflow refuses that input | Denied trust-boundary test |

Use the same representative change set for cold and warm trials. Record download, unpacking, installation scripts, compilation, and cache transfer separately. A large archive can cost more to transfer than it saves. Do not disable lifecycle scripts merely to improve the chart unless the dependency and security owners establish that the resulting installation is valid.

If a cached and clean run disagree, preserve the manifests and relevant evidence, quarantine the suspect entry, and restore the clean path. Treat the disagreement as a correctness failure even if the cached run is faster. The done-when evidence is equivalent required behavior under documented inputs and trust conditions, with a working bypass control.

10. Accept sharding only when the slow path improves

The test owner selects a repeatable suite and records test duration, shared setup, fixture isolation, and external dependency limits. Splitting by file count rarely predicts runtime when one file performs expensive integration work. Balance using observed work and retain a rule for new or unclassified tests so they are not silently omitted.

Use another hypothetical fixture: a nine-minute suite becomes three shards with a two-minute setup on each shard. If their test work takes three, three, and five minutes, the critical shard lasts seven minutes, before queueing and result collection. The change has not produced a three-minute suite, and total runner consumption can increase. Measure both elapsed useful feedback and total resource use.

Prove isolation by running shards concurrently and in different orders. Give mutable databases, ports, accounts, and temporary directories distinct scopes. Deliberately fail one shard and confirm the aggregate gate cannot pass using stale results from an earlier run. Record the exact commit and shard manifest with every result so a missing shard remains visible.

When parallelism causes a dependency to throttle, reduce concurrency or replace that dependency with a reviewed test boundary where appropriate. Do not hide integration coverage by substituting a mock without the test owner's agreement. A shard experiment is accepted only when the relevant feedback distribution improves within the cost and coverage guardrails.

11. Review the experiment with comparable evidence

The delivery lead assembles a comparison table by change class, runner class, cache state, and first-attempt outcome. Include unsuccessful, retried, and canceled runs with their reasons. If the candidate period contains only small documentation changes while the baseline contains release builds, the apparent improvement cannot be attributed to the optimization.

Use enough observations to cover the bottleneck's recurring conditions and disclose the sample size. Keep uncertainty visible when a trial is small or the workload changed. Report early defect feedback, artifact-ready time, release failures, runner consumption, and investigation effort separately. One favorable average does not justify an unrelated coverage reduction.

The service and test owners decide whether to keep, narrow, or reverse the change. Attach the clean-build rehearsal, any cache or test-selection exclusions, and the tested release recovery procedure. A configuration revert affects future runs; identify artifacts or deployments already produced under a faulty configuration before closing the incident.

The next action is a single bounded experiment with a preserved baseline and a named stop authority. For shared-template rollouts across many teams, continue with CI/CD at Scale. Use the CI/CD and observability service for implementation scoped to the workload and its operating constraints.

12. Reconcile failures and close the work

On a cache anomaly, restore the clean path and compare artifacts. On missing test coverage, restore full validation and identify releases that relied on the faulty selection. On runner compromise, isolate the affected pool and follow security containment procedures.

On a release failure, halt further promotion, capture the deployed digest and migration state, and use the preselected recovery path. Reconcile external operations and queued work before resuming. Do not automatically rerun an entire release if a stage may already have committed its effect.

"The baseline includes unsuccessful runs, queueing, retries, and representative change types.", "Cache compatibility and clean fallback have been tested without relying only on a lockfile.", "Selected tests have a dependency model, full-path fallback, and accountable coverage owner.", "Privileged jobs do not trust executable input from lower-trust validation paths.", "The deployed artifact is tied to source, checks, configuration, and release approval.", "Application, data, and external-effect recovery were rehearsed for the selected change class.", "The experiment record reports measured results and limitations without projected savings presented as fact." ]} />

The output is a safer, measured delivery path and a reusable experiment record. Publication review of this guide is not approval to change an organization's release controls; the relevant service, security, and data owners must accept the implementation.