CI/CD at Scale

Scale delivery with versioned pipeline contracts, isolated execution, trustworthy test selection, artifact promotion, and rehearsed multi-team release recovery.

trigger="Several teams depend on shared delivery infrastructure, and inconsistent gates or shared changes are creating release risk." owner="The delivery platform owner, with service owners accountable for their production changes." participants={["Service engineering leads", "Security owner", "Test owners", "Release operator", "Platform on-call", "Risk or compliance owner where applicable"]} prerequisites={[ "An inventory of repositories, release paths, environments, dependencies, and accountable owners.", "Baseline queue, execution, failure, and recovery evidence from representative pipelines.", "A supported pilot cohort, isolated test environment, and retained prior pipeline versions." ]} outputs={[ "A versioned delivery contract and an explicit exception process.", "A tested shared-template release with evidence from representative consumers.", "Service release records linking artifact, configuration, data compatibility, and recovery." ]} doneWhen={[ "Pilot teams can use the platform without hidden manual permissions or undocumented gates.", "Untrusted jobs cannot obtain production credentials through runners, caches, or artifacts.", "Test selection retains coverage or falls back when dependency evidence is uncertain.", "Operators can stop a faulty template rollout and recover a stateful service release." ]} />

Scale the delivery contract, not just the runner fleet

A shared pipeline change can break many services at once. More runners may relieve a queue while leaving fragile templates, unclear approvals, and unsafe release credentials untouched. At scale, the platform itself is a production dependency with users and an incident boundary.

This playbook addresses multi-team governance, reusable delivery contracts, and release coordination. Use CI/CD pipeline optimization for the measured critical path of one pipeline. Neither guide sets a universal build-time or deployment-frequency target.

Start with a representative cohort: a stateless service, a stateful service, and a service with significant external dependencies if those exist in your estate. Choose by risk and variety, not an arbitrary number of teams.

1. Map responsibilities and existing release paths

The platform owner inventories who can change templates, administer runners, publish artifacts, deploy to each environment, and override a failed gate. Include manual releases and emergency access. A diagram of the intended pipeline is incomplete if production still has an undocumented alternate path.

| Decision | Accountable owner | Evidence | | --- | --- | --- | | Shared template behavior | Delivery platform owner | Versioned contract and consumer tests | | Service-specific checks | Service engineering lead | Risk-to-test mapping and results | | Credential and trust boundary | Security owner | Permissions and denied-access tests | | Production go or stop | Release owner | Predefined health and compatibility gates | | Gate exception | Named risk owner | Scope, expiry, mitigation, and audit record | | Shared platform incident | Platform on-call | Containment and restoration runbook |

Measure queue time separately from execution. Record reruns, infrastructure failures, time waiting for approval, and recovery exercises. Segment by workload and repository instead of comparing unrelated teams through a single leaderboard.

Gate: each release path has an owner and a support route. Orphaned privileged paths require explicit remediation before wider platform adoption.

2. Publish a small, versioned pipeline interface

The platform team defines inputs, outputs, required checks, permissions, supported runtimes, and failure behavior. Keep service-specific build logic near the service where practical. A reusable template should expose a supported interface, not require teams to copy hidden implementation details.

Treat template revisions like dependencies. Publish changes, migration notes, compatibility expectations, and a supported prior revision. In GitHub Actions, reusable workflows have defined input and secret interfaces, and a commit SHA provides a stable reference. Confirm platform-specific behavior in the reusable-workflow documentation.

Do not move every consumer to a mutable branch and call the rollout complete. Test a candidate revision against the cohort, then expand with an observable adoption record. Security fixes may require accelerated adoption, but they still need an owner, evidence, and an exception path for consumers that cannot update immediately.

3. Separate execution trust before optimizing capacity

The security and platform owners classify jobs by the code they execute and the authority they need. Pull-request code, third-party actions, generated scripts, and downloaded artifacts are inputs to a trust decision. Repository location alone does not make them safe.

Keep production credentials out of untrusted build jobs. Scope deployment identity to the environment and action. Review runner persistence, network access, artifact permissions, and cache read/write boundaries. A privileged job must not execute a script supplied by an untrusted earlier job merely because it arrived as an artifact.

GitHub's secure-use reference covers least-privilege tokens, untrusted input, third-party action pinning, and related risks. Apply those controls to the actual workflow; adding a scanner does not establish the boundary.

Allocate runner pools and concurrency around workload and trust, then test noisy-neighbor behavior. Interruptible workers suit restartable work only when retries, workspace cleanup, and job deadlines have been tested. Preserve enough capacity and access for emergency releases during ordinary queue saturation.

Gate: an intentionally hostile test job cannot read production secrets, publish trusted release artifacts, or poison a privileged consumer's inputs.

4. Select tests from evidence, not optimism

The test owner maps checks to defects and contracts they protect. Full-suite testing on every commit is not inherently an anti-pattern. It can be the clearest safe choice when the suite is affordable or dependency selection is incomplete.

Affected-only testing requires a trustworthy graph that includes generated code, shared configuration, toolchain changes, runtime dependencies, and cross-service contracts. Unknown relationships should trigger a broader fallback. Compare selected results against full runs during adoption, and retain scheduled or pre-release full verification according to risk.

When a selected set misses a defect, preserve the failing example, correct the dependency model, and review similar exclusions. Do not simply add another filename rule until the failure disappears.

Flaky tests need diagnosis, not deletion for a greener dashboard. A quarantine record specifies the risk covered, owner, deadline, replacement protection, and conditions for returning the test. A critical protection may have no acceptable quarantine path. Reruns can collect diagnostic evidence, but repeated success must not erase the original failure from the report.

5. Promote artifacts and coordinate compatibility

The release owner promotes the same verified artifact through environments using its digest or equivalent immutable identity. Record configuration separately and validate it for the destination. Rebuilding at each stage can change dependencies and invalidate earlier evidence.

Cross-service changes need a compatibility plan. Record which consumer and provider versions can coexist, rollout order, and how mixed-version traffic behaves. Prefer additive transitions where feasible: introduce a compatible interface, migrate consumers, verify adoption, then remove the old behavior in a separately approved change.

Stateful releases require special treatment. A feature flag can stop a code path without undoing database writes, emitted events, emails, or purchases. An application rollback can fail against a destructive schema migration. Record what can be reverted, what persists, and what requires forward repair or business reconciliation.

Manual approval is appropriate when required by risk, contract, or separation of duties. Remove an approval only when its purpose is replaced by adequate evidence and authorized policy, not because all manual gates are considered immature.

6. Release shared templates in controlled cohorts

1. Platform engineering runs contract tests. Exercise supported consumers and negative security cases before offering a template revision.
2. Pilot service owners rehearse real releases. Include a failed deployment and the service-specific recovery path, not only a successful build.
3. Each consumer records adoption. Capture the template revision, artifact identity, exception status and accountable recovery owner.
4. The platform owner retires the old revision after evidence. Confirm scheduled and infrequent consumers have migrated or have a time-bounded exception. Retain the recovery material needed by running services.

The platform owner defines stop signals before rollout: unexpected permissions, missing required evidence, incompatible outputs, or a meaningful regression against the cohort's baseline. Choose observation windows that include the relevant jobs and release paths. A universal canary percentage is not sufficient for a template used unevenly across teams.

If the template fails, freeze expansion and notify affected owners. Restore a supported revision for future runs. Review queued and in-progress jobs before cancellation; a deployment interrupted midway may require recovery rather than a blind rerun.

Classify already-produced artifacts and deployed releases separately. A safe template rollback does not establish that an artifact built by the faulty version is trustworthy. Rebuild or reverify affected outputs as required, with deployment owners deciding remediation for running services.

7. Exercise recovery under a realistic failure

The platform and service operators rehearse a broken shared dependency, unavailable runner pool, and failed stateful release in a controlled environment. Confirm that the prior artifact exists, permissions still work, configuration is recoverable, and data compatibility is understood.

Start the recovery timer at the failed health signal, not after someone finds the correct runbook. Record detection, decision, action, and business recovery separately. A successful deployment command is not proof users have recovered.

For a data-affecting failure, stop harmful work, identify affected records or events, and reconcile the business outcome. Restoring a database snapshot can erase valid concurrent writes, so use only an approved recovery plan with a defined data-loss boundary. Do not attach a generic restore command to every rollback.

8. Build a consumer conformance kit for the shared template

The platform team creates a small set of maintained example consumers that exercise the delivery contract. Include the supported runtime families, artifact types, permission models, and stateful release paths found in the inventory. A template passing its own unit tests does not establish that the services consuming it will still receive the expected outputs or gate names.

For an illustrative runtime upgrade, retain the prior template revision and run the same consumer commits through both revisions. Compare declared inputs, generated metadata, artifact availability, required test reports, and deployment authorization. A changed artifact digest may be expected when the toolchain changes, so the service owner must compare behavior and provenance rather than demanding identical bytes without a reproducible-build contract.

| Conformance case | Required result | Owner deciding failure | | --- | --- | --- | | Supported consumer inputs | Documented outputs and check names remain available | Platform owner | | Missing mandatory input | Actionable failure before deployment authority is granted | Platform owner | | Denied production identity | Build can report results but cannot deploy | Security owner | | Failing required test | Artifact cannot advance through the release gate | Service test owner | | Prior template restored | Supported recovery consumer still operates | Service release owner |

Make each consumer fixture discoverable alongside its protected contract. If a service needs an exception, record the unsupported assumption and a migration owner rather than copying the template into a private fork with no upgrade path. The conformance kit becomes a release artifact for the platform: template revision, consumer commits, test environment, results, and unresolved exclusions.

Gate: the representative consumers pass, denied paths remain denied, and the platform owner can identify which untested consumers need additional review. The kit limits uncertainty; it does not authorize claiming that every service is compatible.

9. Verify production identity at the actual gate

The security owner documents which execution identity may deploy which artifact to which environment. With GitHub Actions federation, the OpenID Connect reference describes subject and audience claims that cloud trust policies can validate. Token issuance alone is not permission to assume a privileged role. Test the cloud-side conditions against the actual workflow context and provider capabilities.

Create an authorized negative-test matrix: an unapproved repository, wrong environment, unexpected branch or workflow, and mismatched audience. Confirm each is denied without printing usable tokens into test evidence. Then run the approved path and verify that its credentials cannot modify a different environment or unrelated resource. Test the policy after changes to reusable workflows and repository ownership settings.

Keep environment gates and runner isolation separate. GitHub's deployment and environment reference documents protection rules and environment-secret access, with availability depending on the configured product and plan. An environment approval does not turn a persistent self-hosted runner into an isolated execution boundary. Verify both controls instead of using one as evidence for the other.

An emergency path needs the same clarity. Name who may invoke it, which checks remain mandatory, which authority is temporary, and how the action is recorded and reviewed. Rehearse access while the normal pipeline is unavailable. The output is a tested authority map that lets responders restore service without granting every repository a permanent bypass.

10. Exercise shared capacity without starving recovery work

The platform operator runs a controlled burst of representative validation jobs while a service owner attempts an authorized recovery deployment. Measure queue wait, startup, execution, artifact transfer, and permission-service availability separately. A reserved runner pool is ineffective if both ordinary and emergency releases depend on the same saturated artifact store or unavailable identity service.

Set concurrency by trust class and workload limits. A large repository should not be able to occupy all privileged deployment slots through ordinary pull-request activity. Conversely, strict global serialization can delay unrelated service recovery. Identify the shared resource being protected and choose the narrowest useful concurrency scope. The acceptance evidence includes the behavior of queued work and the point at which operators can pause new admissions.

Test interruption at distinct stages. A canceled lint job can generally be restarted from source. A canceled artifact upload may require integrity verification. A canceled schema migration or partially completed deployment needs a state-aware recovery decision. Preserve operation and artifact identities so the operator can determine which effects occurred before cancellation.

Record the cost of reserved capacity, the approved queue objective, and the conditions that justify expansion. Avoid a fleet-wide performance claim based only on one fast repository. The platform decision should account for the slow, infrequent, and dependency-heavy consumers that make recovery difficult.

11. Retire a template revision with traceable release evidence

The platform owner maintains an adoption register with repository, service owner, pinned template revision, last tested release path, exceptions, and the supported recovery revision. Include repositories that run only on a release tag or a maintenance schedule. No recent usage may mean an inactive service, an infrequent release, or a missing integration; resolve that uncertainty before declaring migration complete.

Bind each promoted artifact to its source and build evidence. The SLSA provenance specification describes a build record that can support this traceability. Validate the expected builder and subject artifact under the organization's policy. A provenance document from an untrusted producer, or one describing a different artifact, is not sufficient release evidence.

Before retiring the old template, the service owner rehearses the declared recovery path using the remaining supported mechanisms. Confirm that artifact retention, runner images, permissions, and configuration still support it. Archive enough contract and release evidence to investigate a later defect without depending on a deleted job workspace.

If a faulty shared revision already produced releases, classify each output as unpromoted, promoted but not deployed, or running in an environment. Re-pin future runs, but give deployment owners the affected artifact list and the reason for concern. They decide whether revalidation, rebuild, rollback, forward repair, or incident containment is required. Template recovery and service recovery are separate decisions.

The next action is one conformance-kit run and one recovery exercise with a representative service owner. Bring the failed cases and exceptions to the shared-platform review before adding more consumers.

Reusable delivery and exception records

| Record | Required fields | | --- | --- | | Template release | Revision, contract changes, consumer tests, owner, supported prior revision | | Service release | Source commit, artifact digest, configuration, test evidence, environment | | Compatibility | Consumer/provider versions, schema phase, mixed-version tests, removal gate | | Quarantine | Test, protected risk, failure evidence, owner, deadline, temporary coverage | | Exception | Missing gate, affected scope, risk approver, expiry, mitigation, closure proof | | Recovery | Stop signal, operator, rollback limits, persisted effects, reconciliation evidence |

Store records where service owners can retrieve them during an incident. Avoid making access depend on the failed delivery system alone.

Acceptance checklist and limitations

"Every shared and service-specific gate has a purpose and an accountable owner.", "Templates are versioned, consumer-tested, and released with a supported recovery path.", "Untrusted execution cannot reach production authority through credentials or shared state.", "Affected-test selection has verified dependencies and a full-run fallback.", "Flaky-test quarantine preserves the protected risk or blocks release.", "Artifact, configuration, schema compatibility, and deployment identity are recorded together.", "Feature flags, application rollback, and data reconciliation have distinct tested responsibilities." ]} />

This playbook does not mandate a branching model, CI provider, or fully automatic deployment. Repository topology and organizational size alone do not determine those choices. Security and compliance owners must approve the actual controls, and service owners must validate recovery against their data and dependencies. A successful pilot is evidence for the tested cohort, not permission to skip integrated release review.