Production Security Hardening and Verification Checklist
Prioritize production hardening by exposure and consequence, test each control, and keep recovery, exceptions and compliance evidence separate.
trigger="A production service is launching, a material exposure is found, or its configuration no longer matches the approved security baseline." owner="The service owner owns implementation and availability; the security owner prioritizes exposure and verifies control evidence." participants={["Service owner", "Security engineer", "Platform engineer", "Identity owner", "Data owner", "On-call operator"]} prerequisites={["An owned inventory of services, identities and data", "A threat model and versioned security baseline", "Authorized test scope and a non-production rehearsal path", "Recovery access, configuration history and an exception process"]} outputs={["Prioritized control changes with named owners", "Positive, negative and failure-path test evidence", "A release and rollback record", "An exception register with compensating controls and expiry"]} doneWhen={["The selected controls work under the tested production conditions", "Legitimate access and recovery remain available", "Critical denied actions remain denied during rollback", "Open risk is explicitly accepted by an authorized owner"]} />
Harden one owned service, then verify the result
A checked box is useful only when it identifies the system, version, test and owner behind it. Start with the actual exposed service and its dependencies. A generic scanner score does not establish correct authorization or a workable recovery path.
This playbook implements and verifies controls chosen through security-by-design. It covers production application and platform changes. It is not authorization to scan third-party systems, run disruptive tests or change production without the normal approval process.
Remove default or exposed credentials urgently, but coordinate containment with the service and incident owners. A blanket instruction to change every security setting immediately can lock out responders, break authentication or interrupt data access.
1. Establish scope, exposure and priority
The security owner links each finding to an affected asset, reachable path, required privilege and business consequence. Include public APIs, administration, build runners, object stores, backups, internal services and vendor integrations.
| Priority input | Evidence | How it changes the decision | |---|---|---| | Exploitability | Reachable code path, prerequisites and known exploitation evidence | Raises urgency when the condition exists in this deployment | | Data or action consequence | Sensitive records, privileged operations and affected tenants | Determines containment and review requirements | | Existing controls | Tested isolation, authorization and monitoring | May reduce exposure but needs verification | | Change risk | Compatibility, outage and data-recovery constraints | Shapes the rollout and fallback | | Obligation | Applicable policy, contract or legal requirement | Assigned to the responsible security, audit or legal owner |
A vulnerability severity score is one input, not a universal remediation deadline. Record the local target, interim containment, owner and escalation condition. If exploitation is suspected, invoke the incident-response playbook rather than treating the issue as a normal patch ticket.
Gate: the service owner agrees the first bounded change and the security owner agrees how its effect will be demonstrated.
2. Protect identity without breaking the sign-in flow
Inventory administrative, customer, service and emergency identities. Remove shared administrative accounts where attribution is required. Use strong authentication appropriate to the risk, and test enrollment, recovery, lost-device handling and emergency access before enforcing a change.
Use a current identity standard and provider-supported configuration for passwords, authenticators, rate limits and sessions. Do not copy a fixed lockout count or timeout from an unrelated product. Attackers can use aggressive lockout rules to deny service to legitimate users.
For browser sessions, use secure transport and appropriate HttpOnly and Secure cookie attributes. Choose SameSite from the actual sign-in and cross-site flow:
| Policy | Relevant behavior | Regression test | |---|---|---| | Strict | Excludes the cookie from cross-site requests | Entry from an external link and federated return behavior | | Lax | Allows qualifying top-level safe-method navigation | Sign-in redirect and authenticated landing page | | None with Secure | Allows cross-site use where the browser permits it | Required cross-site flow plus separate CSRF protection |
Do not assume Lax supports every cross-site POST callback or that None bypasses browser privacy restrictions. Keep anti-CSRF and protocol validation controls. The OWASP session guidance explains cookie attributes and their limitations.
Test session renewal, logout, account disable, stale permissions and replay of a revoked session. Record how long revocation takes across caches and services.
3. Verify authorization and secret handling
The application owner tests access at the resource and action boundary. A valid login must not permit reading another customer's record, editing protected fields or invoking an administrative action.
Create negative cases using separate test identities and tenants. Cover direct IDs, list filters, exports, background jobs, support access and cached responses. Confirm denial at the server, not only hidden buttons in the UI.
The platform owner inventories secrets by consumer and environment. Prefer managed retrieval or short-lived workload identity where supported. Keep build credentials separate from runtime access, and exclude credentials from source, logs, images and client bundles.
If a secret was exposed, revocation or rotation is the containment action. Deleting it from the latest source revision does not invalidate copies already obtained. Preserve incident evidence under restricted access, test the replacement and prove the old credential no longer works. Plan overlap only where needed for continuity, and end it explicitly.
Gate: allowed operations still work, prohibited operations fail, and responders retain an audited recovery path without using the compromised credential.
4. Stage network and browser-policy changes
Restrict ingress and egress to required destinations and services. Test name resolution, certificate retrieval, identity services, telemetry and recovery dependencies before enforcing a deny policy. A private network alone does not establish application authorization.
Use a maintained TLS configuration supported by the client population and platform. Test certificate renewal and expiry alerts. Introduce HSTS deliberately after verifying HTTPS and subdomain readiness; long-lived browser state can make a mistaken policy difficult to reverse.
Build Content Security Policy from the application's resource and execution needs. Observe a report-only policy where appropriate, then enforce a tested policy. Do not add broad wildcard exceptions simply to clear browser errors.
For server-side URL fetching, validate permitted schemes and destinations, including redirects and the address actually reached. Account for IPv4, IPv6, DNS changes and internal or metadata endpoints. Apply network-level restrictions as another layer. Blocking one textual IP address is not a complete SSRF defense.
Record the exact rule change and affected dependencies. Verify both the intended denial and the legitimate business journey.
5. Harden workload configuration against a named baseline
The platform owner chooses the baseline for the deployed version and environment. Review differences between managed-provider defaults, cluster configuration and workload settings. Do not assume every Kubernetes cluster starts with public administration, root workloads or broad administrator bindings.
For Kubernetes, use the Pod Security Standards as a versioned reference. Test non-root execution, capability restrictions, filesystem permissions and required volume access for the actual workload. Some workloads need a narrow exception; document it instead of disabling the baseline for an entire environment.
Check network-policy enforcement by the installed networking implementation. Test allowed and denied traffic, including DNS. Verify service-account permissions, token exposure, admission behavior, secret access and audit coverage.
Minimal images can reduce unnecessary packages, but they still require patching, provenance and a debugging plan. Pin deployed artifact identity and test upgrades. A digest identifies content; it does not prove the content is trustworthy.
6. Connect supply-chain artifacts to release decisions
Track the source revision, build inputs, dependency versions, artifact digest and deployed environments. Protect the identities allowed to produce and deploy artifacts.
NIST SSDF 1.1 practice PS.3.2 addresses component provenance and gives an SBOM as an example. An SBOM can support that practice, but generating a file alone does not establish SSDF conformance or satisfy every customer obligation. Read the actual practice in NIST SP 800-218 and map required evidence to the applicable scope.
Keep component inventory distinct from build provenance. The SLSA provenance specification describes information about how an artifact was produced. Verify the expected producer and build policy, not merely that a signature exists.
Run relevant source, dependency, secret and image checks at points where they can prevent or detect risk. Triage findings against deployed use and exposure. Record exceptions with a compensating control and expiry. Popularity, recent commits or a clean scan cannot by themselves establish that a dependency is safe.
7. Exercise detection, recovery and the rollback boundary
Generate an authorized test event and follow it from application to alert to responder. Capture identity, action, resource reference, decision, policy version and outcome without unnecessary payloads or secrets. Protect audit storage from ordinary application modification.
Test backup restoration and key access, not only backup-job completion. Use an isolated target and validate business data before considering it a recovery source.
| Hardening failure | Immediate response | Recovery gate | |---|---|---| | Identity policy locks out legitimate users | Use audited recovery access and stop expansion | Required sign-in works; prohibited access remains denied | | Network policy blocks a dependency | Apply a scoped, reviewed correction | Allowed flow works and forbidden flow still fails | | Patch breaks the service | Use a compatible known build plus risk containment | Service and security owners accept the temporary exposure | | Rotation breaks consumers | Repair consumer adoption or issue a safe replacement | Old compromised credential remains revoked | | New system has accepted writes | Fence and reconcile before data failback | Authoritative state and consistency verified |
Do not promise that every security change can be rolled back. HSTS, credential revocation, data migration and external effects have different recovery behavior. Record the boundary before deployment.
8. Separate testing evidence from compliance conclusions
Choose security-test scope, cadence and depth from risk, change rate and applicable obligations. Authorized penetration testing may be valuable, but do not present an annual test as a universal prerequisite for every SOC 2 engagement.
The security owner and auditor determine the relevant controls and evidence. Use the AICPA's SOC resources for the assurance framework. An engineering checklist, scanner result, SBOM or penetration-test report is evidence with a defined scope, not a compliance certificate.
The same boundary applies to privacy, healthcare and payment requirements. Applicability depends on roles, data, processing and contracts. Refer those decisions to the responsible legal, privacy and audit owners rather than inferring them from the presence of a user or a payment integration.
9. Rehearse a service-credential rotation end to end
Use an illustrative service that reads a secret to call a restricted downstream API. The platform owner inventories every consumer before changing it: active application instances, scheduled jobs, recovery tooling, build automation, and any long-lived worker that caches credentials. The security owner identifies the required privileges and whether the current credential is suspected compromised. That distinction changes whether temporary overlap is acceptable.
For an ordinary planned rotation, create the replacement through the approved secret-management path with no more authority than the service requires. The application owner tests it against a sandbox or bounded authorized operation. Confirm both an allowed action and a forbidden action. A successful connection proves authentication, not least privilege. Record the credential identifier or version, never the secret value, in the change evidence.
Deploy to a small, explicitly identified consumer cohort. Check successful business operations, authentication failures, secret-refresh behavior, and unexpected retries. Then exercise a worker restart and a scheduled job so adoption is not inferred only from a warm process. Expand only when the owner can account for consumers and the operator can stop the rollout without leaking the replacement.
| Rotation checkpoint | Evidence produced | Block expansion when | | --- | --- | --- | | Replacement created | Scoped identity and secret version | Privileges exceed the approved operation | | Consumer adoption | Runtime version and successful allowed call | Hidden consumers or refresh failures remain unexplained | | Old credential revoked | Authorized denial test using the old credential | Any route still accepts the revoked authority | | Recovery exercised | Safe replacement or repaired consumer works | Recovery depends on re-enabling a compromised credential | | Record closed | Consumer inventory and residual exceptions | Scheduled or recovery consumers remain untested |
After revocation, distinguish invalid-credential failures from unrelated provider outages. Avoid blind retries that flood the dependency while the service cannot authenticate. Use a bounded unavailable or pending state and escalate with the affected consumer identity. If a replacement fails, repair adoption or issue another safe credential under the incident owner's direction. Do not treat restoring the compromised credential as routine rollback.
For suspected compromise, incident-authorized containment may require revocation before continuity is fully restored. The incident owner accepts that tradeoff and coordinates affected services. Preserve evidence, identify possible use of the exposed authority, and follow the incident process for wider scope. This rehearsal is an operational template, not a universal secret lifetime or an instruction to keep an exposed credential active until every consumer is ready.
10. Test a federated sign-in policy as a complete journey
The identity owner draws the actual browser journey before tightening cookies: application entry, identity-provider redirect, callback method, correlation validation, session creation, and the first protected request. Identify which cookie carries the application session and which temporary state supports the sign-in exchange. Their purposes differ, so copying one SameSite value onto both without testing can break the protocol.
Use an approved browser and identity-provider test matrix. Include a fresh session, an existing session, entry from an external link, the configured cross-site callback, cancellation, an expired exchange, and two concurrent sign-in attempts. Record browser version, callback method, policy revision, and observed acceptance or rejection. Do not place tokens or cookies in screenshots or broadly accessible logs.
The required positive test is that a legitimate journey returns to the expected account and intended application location. The negative test is that altered, expired, or mismatched protocol state cannot create a session. Also verify logout and account disable according to the system's documented revocation behavior. A login that works after weakening protocol validation is a failed hardening change, even if support tickets disappear.
On failure, stop the cohort expansion and diagnose the exact missing cookie or rejected protocol check using protected diagnostics. Use a scoped, reviewed correction consistent with the identity provider's supported flow. Do not disable CSRF protection, accept arbitrary return destinations, or globally weaken cookie policy to clear a browser error. The output is a versioned journey record that explains why each cookie policy was chosen and which flows it was tested against.
11. Keep exceptions from becoming an invisible baseline
An exception should identify the asset, failed control, reason a safe fix is not yet available, compensating protection, accountable risk owner, and expiry. Name the event that forces earlier review, such as new exploit evidence, a changed public route, a dependency upgrade, or expansion to more sensitive data. An expiry date without an owner or escalation path is only a reminder.
The service owner compares deployed configuration with the reviewed baseline after releases and material platform changes. Investigate drift before automatically overwriting it: the difference may be an emergency containment change that needs reconciliation, not an accidental setting. Re-run the relevant allow, deny, and recovery tests against the resulting configuration and attach the new evidence.
Close a hardening item only when the intended control is effective and the business journey remains usable under the tested conditions. Keep failed checks and accepted exceptions visible in the handoff. The next review should start from those unresolved boundaries, not reset the checklist to green because the deployment succeeded.
Reusable control and exception record
Asset, environment and version:
Control ID and baseline source/version:
Exposure and consequence:
Implementation owner and approver:
Allowed, denied and dependency-failure test cases:
Evidence location, date and result:
Deployment scope and stop signals:
Recovery procedure and irreversible effects:
Residual risk and compensating control:
Exception owner, expiry and review trigger:"The scope and test authority are explicit", "Identity and cookie changes pass real sign-in and recovery flows", "Object-level and cross-tenant denials are verified", "Exposed credentials are revoked, not only removed from source", "Network and workload controls preserve required dependencies", "The deployed artifact and component evidence are traceable", "Detection reaches an accountable responder and restoration was exercised", "Exceptions and compliance conclusions remain with authorized owners" ]} />
Limitations and next action
This guide cannot certify a deployment or supply a universal baseline for every runtime. Verify platform-specific behavior and maintain a versioned control record. Repeat the relevant tests when identities, dependencies, data scope or runtime versions change.
Bring unresolved control failures and their evidence to security and compliance engineering.