Which Credential Source Does This EKS Application Select?

Trace one pinned Python application's credential selection before changing EKS Pod Identity permissions. Separate SDK source, running Pod, caller and resource...

Primary sources checked 2026-10-09

Start with the client that made the request

An EKS Pod Identity association can exist while an application continues to use another credential source. Diagnose the exact process and client before changing a role policy. An earlier source can satisfy the SDK, an explicit client can bypass its ordinary chain, or the observation can belong to a different Pod. A successful object read establishes neither the selected provider nor the intended role. Preserve those as separate observations.

The output of this guide is a credential-source diagnostic record for one Python application. It connects the deployed image, client construction, running Pod, association, provider, caller and harmless resource request. It can support an owned decision to retain the current path, correct an application configuration or investigate a missing agent path. It does not authorize a change, certify isolation or provide a general IRSA-to-Pod-Identity migration plan.

The selected teaching context is Linux on EC2-backed Amazon EKS, one account, an ordinary non-host-network application Pod and Python Boto3 1.43.110 with botocore 1.43.110. Those were the current published package versions checked on 9 October 2026. The fixture verifies the exact pins instead of silently exercising a newer SDK. The fictional agent record uses upstream v0.2.1 and an image alias, not an observed deployment or an EKS add-on build identifier. No agent was installed or tested. Real operators must capture their actual distribution, image digest, cluster compatibility and node placement. Package metadata, botocore metadata, agent release.

Fargate, Windows, Auto Mode, cross-account role chaining, custom credential plugins and compromised-node investigation are excluded. Another SDK, explicit profile or custom provider requires a new chain review. All workload identities, observations and outcomes below are fictional. Local tests executed SDK control flow with acquisition replaced by inert stubs and network blocked. They did not call AWS, obtain credentials, evaluate IAM or observe an EKS application.

Record what exists, what ran and what was selected

Begin with the request's application owner, incident revision and collection time. Identify the container image digest, entry point, Python interpreter, installed Boto3/botocore versions and the code that constructs that particular service client. A lock file describes a build intention; retain runtime package evidence from the image being investigated. A sidecar, maintenance shell or administrator CLI can use a different SDK, environment and profile from the application.

Review explicit credential parameters, Session construction and wrappers before inspecting the default chain. Search the controlled source and deployment specification for credential injection mechanisms, without copying their values. Record whether variables exist and where an approved reference comes from. Do not collect a full process environment, shared credentials file, token file, authorization header or debug HTTP dump. Even a read-only diagnostic can disclose secrets if its output is unrestricted.

Pin Pod UID, container identity, start time, image, node, namespace and serviceAccountName. A Deployment's desired template does not establish which revision an old replica runs. During a rollout, collect each relevant replica separately. If the request came from an earlier process, do not attach a replacement Pod's favorable identity record to it. The diagnostic needs a join between the process that made the request and the configuration being reviewed.

The local packet makes that join explicit: the process record names its Pod UID, container identity, cluster, namespace and service account. These must match the supplied running-Pod context, and injection must name that same Pod and container. Changing both a Pod record and its injection does not repair an old process record. Missing or contradictory joins hold the review even when all collection times are recent. These aliases describe supplied evidence, not Kubernetes authentication performed by the fixture.

Separate the evidence sources by their authority. The application owner establishes client construction. Kubernetes metadata establishes the supplied running-Pod context. EKS association readback describes its cluster, namespace, service account and role. The actual SDK observation describes its selected source. Caller evidence identifies the credentials used for that call. Resource evidence describes one operation against one target. Missing access to any record leaves that field unknown; it does not establish that the object or credential source is absent.

Resolve the pinned Python chain without inventing a universal order

Boto3 documents explicit client credentials and explicit Session credentials before its ordinary provider search. Within this pinned botocore resolver, the order is environment, assume-role, web identity, IAM Identity Center, shared credentials, login, process, config file, original EC2 credentials file, Boto configuration, container and instance metadata. The harness obtains this order from the real SDK and asserts it against an independently written expected list. It does not implement its own precedence loop. Boto3 credential guide, pinned resolver source.

The full order prevents two misleading shortcuts. “Environment removed” does not mean container credentials are next. A configured web-identity or shared-file source can still precede them. Conversely, a profile's existence does not mean that profile wins. AWS_PROFILE in the environment and a profile explicitly supplied to Session construction are different cases in this pin. The latter removes the top-level environment provider; the former does not remove it. Review the actual construction instead of diagnosing from one variable name.

An earlier provider returning credentials stops the resolver. A provider returning no credentials lets it continue. An acquisition exception is a separate outcome: this resolver does not catch every error and continue searching. A partial environment credential and a container acquisition error must not be described as equivalent to an absent provider. Web-identity credentials can also defer exchange until first use, so identifying that provider object does not establish a successful exchange or caller. Test provider behavior at the stage being discussed.

Keep the selected source and principal separate. container-role is a botocore method identifier; it is not an IAM ARN or proof that the agent returned the intended role. A role can be reached through more than one supported construction, while two processes can select the same provider type and receive different roles. A chain diagram should show selection and stop/error outcomes, then hand off to the separate identity observation.

Local control-flow model for Boto3/botocore 1.43.110. Explicit credentials bypass the ordinary chain. Otherwise a credential object selects and stops, None continues, and an uncaught exception stops with error. Complete order, profile qualification and refresh limits are part of this figure. All examples are offline and synthetic.

Selection overview, panel 1 of 2. This pinned local model distinguishes absence, selection and an uncaught acquisition exception. A selected deferred object is not a successful exchange. Read the next panel for the complete order, profile qualification and refresh boundary.

Selection detail, panel 2 of 2. The order belongs to this SDK pin. The two inert cases and selected-object refresh do not establish an EKS session lifetime, agent cache or live caller. Advisory failure can retain still-valid credentials; mandatory failure propagates. No credential acquisition occurred.

Match the association to this Pod, not a familiar service-account name

An association is scoped to cluster, namespace and service account. Pod Identity does not use an IRSA-style service-account annotation as its association record. A role mapping for archive-reader in another namespace or cluster does not establish the mapping for this Pod. Retain the exact association ID and dated readback; a friendly console row or copied intended manifest is insufficient. Association guidance, DescribePodIdentityAssociation.

AWS describes Pod Identity injection when a new associated Pod starts. Inspect the actual Pod's container-credential URI reference and authorization-token-file reference, plus the projected mount metadata. Retain only configuration references and presence observations. Do not fetch the URI or open the token to fill this worksheet. An association created after a long-running Pod began is not evidence that its process received the required injection. The date comparison prompts investigation; actual current-Pod metadata remains the evidence. Pod Identity mechanism.

Association changes are eventually consistent. Avoid assuming that an API response and an immediately started replica share the same effective mapping. Use a separately authorized controlled replacement only when required and safe; this guide supplies no restart or rollout command. Capture the replacement UID and process revision, and keep old observations as history rather than overwriting them. A successful association update does not resolve the application's existing cached selection.

IRSA has a different path: its web-identity token is exchanged through STS. If established IRSA remains earlier in this application's chain, a new Pod Identity association does not by itself replace it. Retaining that working arrangement during diagnosis can be preferable to changing several controls at once. Choose any migration under the workload-identity transition framework, with its separate retirement evidence, rather than disabling a known path solely to force a new one to win.

Investigate agent transport only when the selected path needs it

Pod Identity uses the container provider to contact the node-local agent; the agent retrieves credentials through EKS Auth. Agent authority is distinct from the eventual application role. Review the exact agent on the application's node, its image and configuration, and its node-role permission for eks-auth:AssumeRoleForPodIdentity. An agent elsewhere in the cluster being healthy does not establish this node's path. Do not attach broad application permissions to the node role to solve a failed application read. Agent prerequisites.

When a proxy is configured, review the documented link-local proxy bypass for the agent addresses. Separate failure to reach the local agent from its failure to reach EKS Auth, and both from denial by the target service. For a cluster without outbound internet, AWS documents the EKS Auth interface endpoint requirement; IRSA's STS dependency is different. A DNS timeout is not a trust-policy denial. No diagnostic here calls a metadata endpoint or supplies an executable transport probe. Pod Identity considerations, private-cluster dependencies.

Check SDK support before treating missing container behavior as a policy issue. AWS lists Python Boto3 and botocore 1.34.41 as minimum supporting versions. The selected 1.43.110 pair is above that floor, but a minimum-version comparison does not prove the deployed binary, agent configuration or end-to-end request works. Record exact actual versions rather than writing “latest.” Do not copy the AWS CLI's chain into a Python application diagnostic. Supported SDK versions.

If an earlier provider already won, an agent-path failure may be a separate readiness issue, not the cause of the observed caller. Preserve it without changing the causal explanation. Repairing the agent cannot displace a selected environment source in the default resolver. The next decision comes from the source evidence, not whichever dashboard has the most visible warning.

Observe selection, caller and permission without exposing credentials

Use an application-owned, approved diagnostic path scoped to the actual client. Boto3's Session API exposes a credentials object, and this pin labels its method. That observation may invoke acquisition; it is not a harmless permission-free metadata read. Review it before use and never serialize the object or any key, token or frozen credential fields. For lazy providers, record selection separately from successful use. Session reference.

A Session observation alone can miss credentials passed directly to a client. The fixture inspects a pinned private signer field solely to prove that counterexample offline. Private SDK fields are not a stable production logging interface. Prefer controlled application instrumentation whose construction and credential binding the owner has reviewed. If the process cannot expose a safe source observation, mark actual selection unknown and collect the missing evidence under a new authorized test. Do not guess from injected variables.

For caller evidence, an independently constructed STS client can itself select a different source. Correlate any proposed GetCallerIdentity observation with the same reviewed credential path and process revision used for the resource client. AWS documents that this operation returns the calling entity and requires no permission, including the documented explicit-deny behavior. It is therefore not a resource-permission acceptance test. Network availability and credential validity still affect whether a call can be made. Caller identity API.

Record a credential-path alias and an explicit reviewed binding from client identities to that path and process revision. The fixture permits distinct STS and resource client identities only when both appear in this supplied shared-path binding. Source, caller, resource and refresh observations must name that same path. Resource request and refresh must also name the exact application client whose source was observed. A second client in the same process, or the same role obtained through another path, is not interchangeable evidence. This intentionally bounded packet reviews one application client and one shared credential path at a time; collect another packet for another resource client or path. The binding is an owner-review declaration, not proof that two real clients share a credential object.

Record a separately approved harmless resource read with its action, target, selected principal, time and actual outcome. An allowed read by an unexpected principal rejects the intended identity result. A denied resource request with unknown principal remains inconclusive. A known expected caller denied the required read needs permission investigation, not a fabricated pass. A meaningful out-of-scope denial test must identify the same caller and the actual authorization reason; a timeout, missing object or unrelated setup error does not establish that boundary.

Account for cached processes and refresh before changing configuration

A running process is not a live view of every configuration edit. In the pinned SDK, Session retains loaded credentials. The fixture selects an inert environment source, changes the stub's later answer and calls the same Session again; its original selection remains. A new process or Session is a different observation and must have a new revision. This explains why removing a source from a deployment template cannot be assumed to change every existing replica immediately. Pinned Session implementation.

Refresh belongs to the selected credentials object. The real RefreshableCredentials test invokes its supplied container-source callback, without rerunning the complete provider chain. In this pin an advisory refresh failure can retain still-valid credentials, while a mandatory refresh failure propagates. Neither case proves automatic fallback to the node role. The fixture advances an inert clock and checks these control-flow outcomes; it does not exercise an agent cache, token rotation or EKS-issued session lifetime. Pinned refresh implementation.

For an authorized environment check, retain source and caller observations around the relevant refresh interval, together with Pod/process and association revisions. Define the observation window from the workload and credential path rather than inventing a universal waiting period. A short successful read can finish before refresh is needed. A restarted process that succeeds says nothing about an older worker still holding an earlier source.

Do not use restart, trust widening or adding a static key as an automatic diagnostic action. A restart can interrupt application work, and a newly available credential can change the selected identity. The application owner must approve recovery behavior and preserve any uncertain business effects separately. When no safe observation path exists, stop at a documented unknown with an owner rather than forcing a green result.

Keep the chosen request path separate from reachable credentials

The normal chain choosing Pod Identity does not establish that another credential source is inaccessible to code in the Pod. AWS warns that containers are not security boundaries. IMDS restrictions and actual node/Pod configuration need separate review. Host-network Pods have documented IMDS access and are excluded from this exercise; that warning does not mean their ordinary supported SDK always selects node credentials. The selected-path question and the reachability question remain different. IRSA isolation guidance.

For this non-host-network cohort, retain the owned IMDS restriction evidence rather than infer it from hostNetwork: false. The classifier holds a broader readiness record when reachability is unknown or declared reachable, even if source/caller/read observations align. That is this guide's conservative review policy, not a claim that the credential request itself failed. Its output always states that it cannot prove isolation.

Three alternatives remain legitimate. Retain verified IRSA while investigating the new path. Correct earlier source configuration through an owned rollout, preserving recovery and then reobserve the default chain. Or deliberately select an appropriate provider in reviewed application code, accepting that the client is no longer explained by the ordinary default-chain assumption. The last option requires supported integration and lifecycle testing; copying temporary values into explicit parameters is not equivalent to a refresh-capable provider. No alternative is authorized by an offline fixture result.

Work the wrong-caller example and the missing-evidence cases

Fictional process A runs in orders as archive-reader. A Pod Identity association points to archive-role, but an earlier environment source is still available. The real resolver harness, supplied with inert answers, chooses env and never invokes the container provider. Separately stipulated caller evidence says legacy-role; an approved-fixture read is stipulated allowed. The correct identity decision is rejection, even though the data arrived. The harness did not obtain that principal or perform that read.

Process B is a separately stipulated replacement with the reviewed image and default client. Earlier sources are absent in its inert SDK case, and container credentials are present. The resolver chooses container-role without invoking the final instance provider. Its evidence packet stipulates matching Pod injection, association, agent-node mapping, archive-role, allowed harmless read, renewed selected source and reviewed IMDS restrictions. The classifier returns DIAGNOSTIC_REVIEW_READY only. Every reference still needs authentication by an owner; no packet field proves an observation occurred.

Change only the association service account to other, and the packet rejects the workload mapping. Remove the current-Pod injection record, and it holds. Supply caller evidence from process A for process B, and it holds. Replace the source with unknown and mark the read denied, and it holds without attributing the denial to policy. Configure the inert container provider to raise while the final instance provider is available, and the real resolver raises rather than selecting that final provider. These counterexamples prevent one successful result from defining all outcomes.

Two additional cross-copy failures matter. Replacing Pod UID and matching injection together while retaining process B's old Pod binding holds. Changing the Pod and association namespace together while keeping the old process also holds. A caller on another credential path holds even inside process B; a request or refresh attributed to another client holds even if that client is listed in the shared-path binding. Conversely, distinct resource and STS clients explicitly bound to path-b are supported. These are consistency tests over declarations, not newly executed SDK or AWS observations.

The fixture uses relative observation integers and a declared 50-unit age limit. They are bookkeeping, not seconds, AWS propagation timing or a recommended production window. Expected outcomes are written in tests independently of the classifier. All results contain authorizesAwsChange: false, provesSdkRuntime: false and provesIsolation: false. The real-SDK tests establish only the pinned local control flow they exercise.

In the companion JSON, use null for a missing or unobserved role in association.expectedRole, caller.role, request.role or refresh.role. Each such value holds the review. A supplied identity alias is a lowercase string matching [a-z][a-z0-9-]{0,79}; matching aliases establish only agreement between declarations, never an authenticated IAM identity. Words such as unknown are not reserved role sentinels: if entered as an alias, they are treated as supplied identities. Do not enter one to represent unavailable evidence. In other fields, the documented unknown outcome or reachability enum retains its separate meaning. Figure labels describing unknown evidence are prose, not JSON role values.

Synthetic Linux EC2-backed EKS record. Solid arrows model the conditional agent/Auth path; desktop dashed arrows are supplied client-path declarations, and dotted lines mean separate reachability review. Mobile retains those meanings in explicit records and text. These joins do not authenticate runtime evidence. Missing or conflicting joins hold; review-ready cannot authorize change or prove isolation.

Evidence context, panel 1 of 2. Workload boundaries express stipulated membership, not enforced isolation. Dashed bindings are owner-review declarations, not proved SDK object equality. The solid agent/Auth route is conditional, not captured traffic; dotted IMDS review is a separate surface. Read the next panel for observation admission.

Evidence admission, panel 2 of 2. The times are fictional relative units. A common process or role cannot replace the explicit client/path join. Missing or conflicting joins hold; known wrong source or role rejects. Review-ready remains a supplied-record assessment with no change, runtime or isolation authority.

Retain a filled and blank diagnostic record

The following filled record is fictional. Aliases and synthetic/ references contain no account, credential or customer data. Keep the entire record together when handing off a held case; omitting the unknown field can change its meaning.

Filled fictional record

Scope and owner

Example review-b; application/platform/security owners are role labels only. Linux EC2-backed EKS, one account, non-host-network Pod; no AWS exercise.

Application and client

image-b, process-b at relative 70 bound to pod-b/container-b in cluster-a/orders/archive-reader; Python Boto3 1.43.110 and botocore 1.43.110; default client, no explicit credentials or constructor profile; synthetic/process-b.

Running Pod

pod-b/container-b, created 60, cluster-a, orders/archive-reader, node-a, hostNetwork false; synthetic/pod-b. No real UID or digest supplied.

Association

archive-role for cluster-a/orders/archive-reader, observation 55; synthetic/association-a. archive-role is a supplied synthetic identity alias, not a missing-value sentinel. Identifier, trust and effective permissions remain stipulated, not inspected.

Injected references

Container URI and authorization-token-file references present for pod-b/container-b at 60; synthetic/injection-b. No endpoint request or token read.

Agent and Auth path

Upstream v0.2.1, agent-image-a on node-a; Auth-path review stipulated at 85 under synthetic/agent-a. No real image, node permission or endpoint tested.

Earlier and explicit sources

All earlier provider answers absent in the B harness case. No client override. This is a stubbed case, not inspection of a running process.

Selected source

container-role for process-b/resource-client-b on path-b at 80; synthetic/source-b. Reviewed binding at 75 lists resource-client-b and sts-client-b on path-b under synthetic/client-binding-b. Inert SDK case agrees; no real application acquisition proven.

Caller and resource

archive-role via process-b/sts-client-b/path-b at 82; approved-fixture get-synthetic-object allowed through process-b/resource-client-b/path-b at 83; synthetic/caller-b and synthetic/read-b. No AWS request made.

Refresh

Same container-role/archive-role for process-b/resource-client-b/path-b renewed at 90; synthetic/refresh-b. Agent/token lifecycle remains unexecuted.

Reachability and denial

IMDS restriction review stipulated at 85 under synthetic/isolation-b. No real isolation or out-of-scope authorization denial demonstrated.

Window and disposition

asOf 100, maxAge 50 relative units; DIAGNOSTIC_REVIEW_READY only. Authenticate evidence and separately authorize any environment test or change.

Blank reusable record

Scope and owner

Incident/revision: ____; application/platform/security owners: ____; account/Region/cluster context: ____; supported/excluded workload: ____; actual versus proposed checks: ____.

Application and client

Image digest/interpreter/SDK package hashes: ____; process revision/start: ____; bound Pod UID/container/cluster/namespace/service account: ____; exact client identity/construction and overrides: ____; reviewed diagnostic path: ____.

Running Pod

UID/creation time/image/node: ____; cluster/namespace/serviceAccountName: ____; hostNetwork: ____; collector/time/reference: ____.

Association

ID/cluster/namespace/service account/role: ____; creation/modification/readback times: ____; trust and permission review references: ____; propagation uncertainty: ____.

Injected references

Matching Pod/container: ____; URI reference and token-file/mount presence only: ____; capture time/revision: ____; missing injection owner: ____.

Agent and Auth path

Actual distribution/version/image digest/node placement: ____; node-role Auth permission evidence: ____; local agent/proxy and Auth endpoint evidence: ____; compatibility gaps: ____.

Earlier and explicit sources

Explicit client/Session credentials: ____; profile selection form: ____; earlier environment/web-identity/file/process sources present or unknown: ____; values excluded from record: ____.

Selected source

Actual process/client/credential-path/provider method: ____; reviewed binding listing client identities, path, process revision, time and evidence: ____; acquisition/deferred/error status: ____; collection method/time/reference: ____; unresolved source owner: ____.

Caller and resource

Expected and observed sanitized principal: ____; companion JSON role fields use null when missing or unobserved, never a word used as a placeholder alias; caller client/path/process and explicit shared-path binding: ____; exact resource client/path/process: ____; harmless action/resource/result/reason/time: ____; unknown or mismatching observation: ____.

Refresh

Same process, exact resource client and credential path or replacement revision: ____; before/after source and caller: ____; refresh observation window/result/reference: ____; retained credential uncertainty: ____.

Reachability and denial

Owned IMDS/other-source reachability review: ____; approved out-of-scope harmless request/principal/authorization reason: ____; untested boundaries: ____.

Window and disposition

Actual collection/recheck window: ____; HOLD or rejected mismatch and next owner: ____; evidence supporting review readiness: ____; separate change authority and recovery conditions: ____.

Reproduce only the offline decision, then resolve the first unknown

The downloadable companion files are sdk_harness.py, diagnostic.py, packet.json, test_diagnostic.py and hash-pinned requirements.txt. The README gives isolated installation and reproduction instructions. Dependency setup uses the public package index; the actual test run blocks sockets, DNS connection setup and external credential processes. It clears ambient environment configuration and uses temporary SDK config paths, with provider acquisition replaced before resolution. Never substitute real tokens or credentials for the inert sentinels.

Download the offline credential-source exercise (ZIP). The six-file package contains the two separate test layers, synthetic packet, pinned requirements and README. Dependencies are not bundled. Use an isolated Python 3.12 environment; dependency installation retrieves public packages, while the tests block network acquisition. No AWS request or real credential is required.

Run the tests in the documented isolated environment, inspect the expected assertions and change one supplied observation at a time. The SDK cases use the real pinned resolver, Session caching, explicit-client binding and refresh object, while substituting acquisition. The classifier separately checks supplied identity and evidence consistency. Passing either layer does not establish the other, and neither proves the behavior of a deployed application. The model cannot authenticate evidence references, inspect a real agent, calculate effective IAM permissions or establish an isolation boundary.

If actual selection is unknown, the next useful action is an approved application-scoped observation, not a broader role policy. If the principal is known and unexpected, contain the mismatch under the service's recovery process before changing credentials. If the expected principal is denied, investigate that action and resource with its authorization owner. For the larger migration and retirement decision, return to the existing identity-transition owner. For a node-image replacement, keep the separate AL2023 application identity admission. Attach this diagnostic record to either workflow without calling the underlying migration complete.