CI/CD at Scale

Scale delivery with versioned pipeline contracts, isolated execution, trustworthy test selection, artifact promotion, and rehearsed multi-team release recovery.

Scale the delivery contract, not just the runner fleet

A change to a shared continuous integration and continuous delivery (CI/CD) pipeline can break many services at once. More runners may relieve a queue while leaving fragile templates, unclear approvals, and unsafe release credentials untouched. At scale, the platform itself is a production dependency with users and an incident boundary.

This playbook addresses multi-team governance, reusable delivery contracts, and release coordination. Use CI/CD pipeline optimization for the measured critical path of one pipeline. Neither guide sets a universal build-time or deployment-frequency target.

Start with a representative cohort: a stateless service, a stateful service, and a service with significant external dependencies if those exist in your estate. Choose by risk and variety, not an arbitrary number of teams.

1. Map responsibilities and existing release paths

The platform owner inventories who can change templates, administer runners, publish artifacts, deploy to each environment, and override a failed gate. Include manual releases and emergency access. A diagram of the intended pipeline is incomplete if production still has an undocumented alternate path.

Measure queue time separately from execution. Record reruns, infrastructure failures, time waiting for approval, and recovery exercises. Segment by workload and repository instead of comparing unrelated teams through a single leaderboard.

Gate: each release path has an owner and a support route. Orphaned privileged paths require explicit remediation before wider platform adoption.

2. Publish a small, versioned pipeline interface

The platform team defines inputs, outputs, required checks, permissions, supported runtimes, and failure behavior. Keep service-specific build logic near the service where practical. A reusable template should expose a supported interface, not require teams to copy hidden implementation details.

Build evidence does not grant deploy authority

Service code uses a versioned template and isolated build. A verified artifact reaches a protected deployment gate with scoped identity and service authority. A direct bypass is blocked. Gate and rollout outcomes enter a release record.

Build evidence does not grant deploy authority

PROTECTED DEPLOYMENT AUTHORITY

Service code

Protected revision

Declared inputs

Shared template

Versioned contract

Consumer tests

Isolated build

No deploy authority

Bounded execution

Verified artifact

Digest / provenance

Compatibility evidence

Deployment gate

Scoped identity

Service authority

Service rollout

Bounded exposure

Recovery owner

Release record

Artifact, gate, outcome

1. Service code enters a versioned shared template and isolated build without deployment authority.
2. The resulting artifact carries a digest, provenance and compatibility evidence.
3. The protected gate verifies the artifact and the scoped deployment identity. A direct code-to-deployment bypass is blocked.
4. The service owner controls rollout and recovery. Gate and rollout outcomes are recorded with the artifact identity.

Reference control boundary. Shared templates produce evidence; scoped identity, environment policy and service authority govern deployment. The release record makes both paths auditable.

Treat template revisions like dependencies. Publish changes, migration notes, compatibility expectations, and a supported prior revision. In GitHub Actions, reusable workflows have defined input and secret interfaces, and a commit SHA provides a stable reference. Confirm platform-specific behavior in the reusable-workflow documentation.

Do not move every consumer to a mutable branch and call the rollout complete. Test a candidate revision against the cohort, then expand with an observable adoption record. Security fixes may require accelerated adoption, but they still need an owner, evidence, and an exception path for consumers that cannot update immediately.

3. Separate execution trust before optimizing capacity

The security and platform owners classify jobs by the code they execute and the authority they need. Pull-request code, third-party actions, generated scripts, and downloaded artifacts are inputs to a trust decision. Repository location alone does not make them safe.

Keep production credentials out of untrusted build jobs. Scope deployment identity to the environment and action. Review runner persistence, network access, artifact permissions, and cache read/write boundaries. A privileged job must not execute a script supplied by an untrusted earlier job merely because it arrived as an artifact.

GitHub's secure-use reference covers least-privilege tokens, untrusted input, third-party action pinning, and related risks. Apply those controls to the actual workflow; adding a scanner does not establish the boundary.

Allocate runner pools and concurrency around workload and trust, then test noisy-neighbor behavior. Interruptible workers suit restartable work only when retries, workspace cleanup, and job deadlines have been tested. Preserve enough capacity and access for emergency releases during ordinary queue saturation.

Gate: an intentionally hostile test job cannot read production secrets, publish trusted release artifacts, or poison a privileged consumer's inputs.

4. Select tests from evidence, not optimism

The test owner maps checks to defects and contracts they protect. Full-suite testing on every commit is not inherently an anti-pattern. It can be the clearest safe choice when the suite is affordable or dependency selection is incomplete.

Running only tests affected by a change requires a trustworthy dependency graph. It must include generated code, shared configuration, toolchain changes, runtime dependencies, and cross-service contracts. Unknown relationships should trigger a broader fallback. Compare selected results against full runs during adoption, and retain scheduled or pre-release full verification according to risk.

When a selected set misses a defect, preserve the failing example, correct the dependency model, and review similar exclusions. Do not simply add another filename rule until the failure disappears.

Flaky tests need diagnosis, not deletion for a greener dashboard. A quarantine record specifies the risk covered, owner, deadline, replacement protection, and conditions for returning the test. A critical protection may have no acceptable quarantine path. Reruns can collect diagnostic evidence, but repeated success must not erase the original failure from the report.

5. Promote artifacts and coordinate compatibility

The release owner promotes the same verified artifact through environments using its digest or equivalent immutable identity. Record configuration separately and validate it for the destination. Rebuilding at each stage can change dependencies and invalidate earlier evidence.

Cross-service changes need a compatibility plan. Record which consumer and provider versions can coexist, rollout order, and how mixed-version traffic behaves. Prefer additive transitions where feasible: introduce a compatible interface, migrate consumers, verify adoption, then remove the old behavior in a separately approved change.

Stateful releases require special treatment. A feature flag can stop a code path without undoing database writes, emitted events, emails, or purchases. An application rollback can fail against a destructive schema migration. Record what can be reverted, what persists, and what requires forward repair or business reconciliation.

Manual approval is appropriate when required by risk, contract, or separation of duties. Remove an approval only when its purpose is replaced by adequate evidence and authorized policy, not because all manual gates are considered immature.

6. Release shared templates in controlled cohorts

1. Platform engineering runs contract tests. Exercise supported consumers and negative security cases before offering a template revision.
2. Pilot service owners rehearse real releases. Include a failed deployment and the service-specific recovery path, not only a successful build.
3. Each consumer records adoption. Capture the template revision, artifact identity, exception status and accountable recovery owner.
4. The platform owner retires the old revision after evidence. Confirm scheduled and infrequent consumers have migrated or have a time-bounded exception. Retain the recovery material needed by running services.

The platform owner defines stop signals before rollout: unexpected permissions, missing required evidence, incompatible outputs, or a meaningful regression against the cohort's baseline. Choose observation windows that include the relevant jobs and release paths. A universal canary percentage is not sufficient for a template used unevenly across teams.

If the template fails, freeze expansion and notify affected owners. Restore a supported revision for future runs. Review queued and in-progress jobs before cancellation; a deployment interrupted midway may require recovery rather than a blind rerun.

Classify already-produced artifacts and deployed releases separately. A safe template rollback does not establish that an artifact built by the faulty version is trustworthy. Rebuild or reverify affected outputs as required, with deployment owners deciding remediation for running services.

7. Exercise recovery under a realistic failure

The platform and service operators rehearse a broken shared dependency, unavailable runner pool, and failed stateful release in a controlled environment. Confirm that the prior artifact exists, permissions still work, configuration is recoverable, and data compatibility is understood.

Start the recovery timer at the failed health signal, not after someone finds the correct runbook. Record detection, decision, action, and business recovery separately. A successful deployment command is not proof users have recovered.

For a data-affecting failure, stop harmful work, identify affected records or events, and reconcile the business outcome. Restoring a database snapshot can erase valid concurrent writes, so use only an approved recovery plan with a defined data-loss boundary. Do not attach a generic restore command to every rollback.

8. Build a consumer conformance kit for the shared template

The platform team creates a small set of maintained example consumers that exercise the delivery contract. Include the supported runtime families, artifact types, permission models, and stateful release paths found in the inventory. A template passing its own unit tests does not establish that the services consuming it will still receive the expected outputs or gate names.

For an illustrative runtime upgrade, retain the prior template revision and run the same consumer commits through both revisions. Compare declared inputs, generated metadata, artifact availability, required test reports, and deployment authorization. A changed artifact digest may be expected when the toolchain changes, so the service owner must compare behavior and provenance rather than demanding identical bytes without a reproducible-build contract.

Make each consumer fixture discoverable alongside its protected contract. If a service needs an exception, record the unsupported assumption and a migration owner rather than copying the template into a private fork with no upgrade path. The conformance kit becomes a release artifact for the platform: template revision, consumer commits, test environment, results, and unresolved exclusions.

Gate: the representative consumers pass, denied paths remain denied, and the platform owner can identify which untested consumers need additional review. The kit limits uncertainty; it does not authorize claiming that every service is compatible.

9. Verify production identity at the actual gate

The security owner documents which execution identity may deploy which artifact to which environment. With GitHub Actions federation, the OpenID Connect reference describes subject and audience claims that cloud trust policies can validate. Token issuance alone is not permission to assume a privileged role. Test the cloud-side conditions against the actual workflow context and provider capabilities.

Create an authorized negative-test matrix: an unapproved repository, wrong environment, unexpected branch or workflow, and mismatched audience. Confirm each is denied without printing usable tokens into test evidence. Then run the approved path and verify that its credentials cannot modify a different environment or unrelated resource. Test the policy after changes to reusable workflows and repository ownership settings.

Keep environment gates and runner isolation separate. GitHub's deployment and environment reference documents protection rules and environment-secret access, with availability depending on the configured product and plan. An environment approval does not turn a persistent self-hosted runner into an isolated execution boundary. Verify both controls instead of using one as evidence for the other.

An emergency path needs the same clarity. Name who may invoke it, which checks remain mandatory, which authority is temporary, and how the action is recorded and reviewed. Rehearse access while the normal pipeline is unavailable. The output is a tested authority map that lets responders restore service without granting every repository a permanent bypass.

10. Exercise shared capacity without starving recovery work

The platform operator runs a controlled burst of representative validation jobs while a service owner attempts an authorized recovery deployment. Measure queue wait, startup, execution, artifact transfer, and permission-service availability separately. A reserved runner pool is ineffective if both ordinary and emergency releases depend on the same saturated artifact store or unavailable identity service.

Set concurrency by trust class and workload limits. A large repository should not be able to occupy all privileged deployment slots through ordinary pull-request activity. Conversely, strict global serialization can delay unrelated service recovery. Identify the shared resource being protected and choose the narrowest useful concurrency scope. The acceptance evidence includes the behavior of queued work and the point at which operators can pause new admissions.

Test interruption at distinct stages. A canceled lint job can generally be restarted from source. A canceled artifact upload may require integrity verification. A canceled schema migration or partially completed deployment needs a state-aware recovery decision. Preserve operation and artifact identities so the operator can determine which effects occurred before cancellation.

Record the cost of reserved capacity, the approved queue objective, and the conditions that justify expansion. Avoid a fleet-wide performance claim based only on one fast repository. The platform decision should account for the slow, infrequent, and dependency-heavy consumers that make recovery difficult.

11. Retire a template revision with traceable release evidence

The platform owner maintains an adoption register with repository, service owner, pinned template revision, last tested release path, exceptions, and the supported recovery revision. Include repositories that run only on a release tag or a maintenance schedule. No recent usage may mean an inactive service, an infrequent release, or a missing integration; resolve that uncertainty before declaring migration complete.

Bind each promoted artifact to its source and build evidence. The SLSA provenance specification describes a build record that can support this traceability. Validate the expected builder and subject artifact under the organization's policy. A provenance document from an untrusted producer, or one describing a different artifact, is not sufficient release evidence.

Before retiring the old template, the service owner rehearses the declared recovery path using the remaining supported mechanisms. Confirm that artifact retention, runner images, permissions, and configuration still support it. Archive enough contract and release evidence to investigate a later defect without depending on a deleted job workspace.

If a faulty shared revision already produced releases, classify each output as unpromoted, promoted but not deployed, or running in an environment. Re-pin future runs, but give deployment owners the affected artifact list and the reason for concern. They decide whether revalidation, rebuild, rollback, forward repair, or incident containment is required. Template recovery and service recovery are separate decisions.

The next action is one conformance-kit run and one recovery exercise with a representative service owner. Bring the failed cases and exceptions to the shared-platform review before adding more consumers.

Reusable delivery and exception records

Store records where service owners can retrieve them during an incident. Avoid making access depend on the failed delivery system alone.

The release-evidence and recovery design framework expands a service release record into an artifact-bound approval packet, with a worked stateful-change example and explicit recovery limits.

Acceptance checklist and limitations

• Every shared and service-specific gate has a purpose and an accountable owner.
• Templates are versioned, consumer-tested, and released with a supported recovery path.
• Untrusted execution cannot reach production authority through credentials or shared state.
• Affected-test selection has verified dependencies and a full-run fallback.
• Flaky-test quarantine preserves the protected risk or blocks release.
• Artifact, configuration, schema compatibility, and deployment identity are recorded together.
• Feature flags, application rollback, and data reconciliation have distinct tested responsibilities.

This playbook does not mandate a branching model, CI provider, or fully automatic deployment. Repository topology and organizational size alone do not determine those choices. Security and compliance owners must approve the actual controls, and service owners must validate recovery against their data and dependencies. A successful pilot is evidence for the tested cohort, not permission to skip integrated release review.