Test Access Revocation in AI Search

Rehearse permission removal across AI retrieval, caches, conversation history and in-flight jobs. Capture boundary-specific denial evidence and stop on disclosure.

trigger="An AI search application checks permissions for new searches, but has not proved what happens after an existing reader loses access." owner="The application owner accountable for preventing unauthorized disclosure." participants={['Identity engineer', 'Retrieval engineer', 'Cache and conversation owner', 'Security reviewer', 'Independent test observer']} prerequisites={['Synthetic documents and identities in an isolated environment', 'An approved revocation contract and timing boundary', 'An inventory of disclosure surfaces and current policy sources', 'A tested stop control and restricted evidence location']} outputs={['A boundary-by-boundary test matrix', 'Baseline access and control evidence', 'Revocation and post-change readback observations', 'Owned disclosure findings and release decisions']} doneWhen={['Every inventoried surface has an evidenced disposition', 'Revoked fixtures are not disclosed beyond the approved boundary', 'Authorized controls remain usable', 'In-flight and stale-policy cases have explicit results', 'The report states coverage and unresolved limitations']} />

Prove the boundary after access changes

Use this playbook to test what an AI search application does when a previously authorized reader loses access to a document, group or tenant. The target outcome is a reproducible record of when the change took effect at each disclosure boundary. A successful identity-provider response is not enough to establish that retrieval, cached answers and stored conversations enforce the new decision.

This is proposed engineering guidance, not a security certification or a claim about a deployed Ampity system. Run the initial exercise using synthetic records and isolated identities. A real revocation can interrupt business work, and a real disclosure can expose confidential information. This article does not authorize either a production permission change or an attempt to access other people's data.

Revocation is also different from deletion. The document may remain valid for other users while one reader loses access. Removing the document from every store could make the test appear successful while breaking the intended permission boundary. The companion access-revocation and cache article explains the problem; this playbook provides the procedure.

Begin with one source family and one supported permission-change path. Expand only after the team can identify the authoritative policy, the observed enforcement points and the evidence used to make the result repeatable. The exercise should reveal missing boundaries, not conceal them with an overly broad “access revoked” status.

The sequence is an exercise plan, not a diagram of a deployed customer system. Each disclosure path needs its own observation. A token-revocation response alone cannot establish document-access revocation.

1. Agree the revocation contract and stop conditions

Owner: application owner with security reviewer. Output: approved exercise contract. Define the subject, resource revisions, environment and exact permission being removed. State which event begins the timing measurement and which event means the application must stop disclosing the protected content. Do not silently substitute token expiry for a promised document-permission change.

Decide how the contract treats requests already in progress. A policy might require checking again before releasing a completed answer, or define a narrower boundary for requests accepted before the change. Document the choice, its consequence and its approval. An arbitrary delay tolerated by a test script must not become the application's implicit security policy.

Set stop conditions: any unexpected tenant, real credentials, an unapproved live destination, missing policy identity or protected fixture text appearing after the approved boundary. Demonstrate the stop control before warming caches or starting delayed jobs. An operator should be able to isolate the affected path without broadening access to investigate it.

Record the permitted evidence retention. Screenshots, model traces and generated answers can become copies of protected data. Synthetic markers make the first exercise safer, but the later operating procedure still needs restricted access, bounded retention and an owner for removing unnecessary test evidence.

2. Inventory every surface that can disclose content

Owner: retrieval engineer with application and storage owners. Output: disclosure-surface matrix. Include search snippets, retrieved chunks, generated answers, source previews, citations, direct document links, downloaded reports, stored conversations and cached results that the application actually exposes. Identify both browser and API paths where they differ.

Trace which identity and policy each boundary uses. A retrieval filter may use current group membership while an answer cache uses a previous authorization decision. A source-preview endpoint might rely on a different access layer. Treat those as separate paths with separate evidence, even if the interface renders them in one panel.

Microsoft's security-filter example for Azure AI Search uses principal strings in query filtering and explicitly distinguishes this from authenticating or authorizing those principals. The application must obtain and enforce the appropriate trusted identity and policy. See Azure AI Search's security-filter pattern.

Mark unknown or unobservable surfaces as gaps. Do not infer that a provider-managed conversation store is safe merely because its UI is hidden. Identify whether the application can return stored content from it and which boundary controls that return. Limit the final claim to surfaces actually inventoried and tested.

3. Prepare fixtures that expose both leaks and over-denial

Owner: independent test observer. Output: fixture manifest and baseline. Create a uniquely recognizable protected document, an authorized control document and a similarly named document in another tenant. Use synthetic content and stable revision identities. The protected marker should be specific enough to detect without resembling a real secret.

Create a reader who initially has access and a reader who never had it. If supported, include an independent reader who retains access after the change. These controls distinguish a correct per-reader revocation from deleting the whole document, breaking retrieval generally or denying everyone indiscriminately.

Prove the baseline through the actual application. Confirm the authorized reader can retrieve the protected fixture, the unauthorized reader cannot, and the control remains available. Save bounded observations with timestamps and configuration identity. If baseline denial already fails, stop and record that defect before testing revocation.

Do not rely only on a keyword query. Include a paraphrased question, a direct source reference and a question that invites the model to synthesize protected facts with public material. The goal is to exercise supported disclosure paths, not to assume exact-match retrieval covers every route to the fixture.

4. Warm the state that could outlive permission removal

Owner: cache and conversation owner. Output: recorded warm-state inventory. Run the authorized requests needed to populate retrieval caches, answer caches and conversation history. Record which state was actually created. If a cache cannot be observed directly, identify the supported evidence that distinguishes a hit from a fresh computation.

Save an existing browser session and an API session if both are supported. Prepare a conversation with the protected document already used as context. Keep the original request identity and synthetic question so the post-change test can exercise the same state, not accidentally start a clean session that avoids the defect.

Start one controllable delayed job that has fetched the protected content but has not yet released its result. Define the pause point and how the observer will resume it safely. The job is a fixture for testing the output boundary, not permission to capture real user content in a debugger.

Separate authentication state from content authorization. A reader may remain logged in while losing document access. Testing only logout can miss that case. Conversely, revoking the entire session is a different exercise with its own expected behavior and should not be conflated with group membership removal.

5. Apply the approved change and record propagation

Owner: identity engineer or policy owner. Output: authoritative change receipt and timing record. Remove the scoped permission through the system's supported path. Capture the change identity and relevant policy version without copying credentials. Record the response time separately from observations at the application boundaries.

RFC 7009 describes OAuth token revocation and acknowledges possible propagation delay between servers. That protocol event is useful evidence when token revocation is part of the exercise, but it does not establish the disposition of application document permissions or existing answer copies. See OAuth token revocation.

Observe the current policy through its supported readback mechanism. If a connector or search index has not yet incorporated the change, record the interval and the affected surface. Do not hide the delay by restarting every component unless restarting is genuinely part of the approved operating procedure.

If the change fails or its scope is uncertain, stop the exercise before interpreting subsequent reads. A denial caused by an unrelated outage does not prove revocation. Preserve the failed change evidence and re-establish the controls before repeating the procedure.

6. Test new requests with existing and fresh sessions

Owner: application engineer with independent observer. Output: post-change request matrix. Repeat protected and control queries using the existing session first, then a fresh session under the same reader identity. Inspect snippets, answer text, source titles and citation destinations. A response can leak protected information without returning the complete source.

Use supported direct-reference paths as well as normal search. A user who knows a source identifier must not bypass the current policy through a preview or download endpoint. Exercise browser and API variants where their handlers differ. Record the identity, policy observation and response disposition for each case.

OWASP recommends validating permissions on every request and protecting static-resource paths as well as application handlers. This supports testing the whole disclosure surface rather than only the search route. See OWASP's authorization guidance.

Verify the unchanged reader and control document throughout. A revoked fixture disappearing because every query fails is not a pass. The independent observer should be able to reproduce both the denial and a permitted result from the same configuration without privileged shortcuts.

7. Challenge caches and previously generated answers

Owner: cache owner. Output: cache-boundary readback evidence. Repeat warmed questions, paraphrases and source references after the approved boundary. Determine whether an old answer is rechecked against current permissions before it is served. Do not assume a short time-to-live prevents a leak within that interval.

Inspect the cache's identity and policy scope. A shared entry keyed only by question text can create a different defect from a reader-specific entry carrying an obsolete authorization decision. Test both cross-reader separation and same-reader revocation where the implementation supports these states.

RFC 7662 explains the security-performance trade-off when token introspection results are cached, including a stale-authorization window after revocation. Apply that lesson carefully: it describes authorization information caching, not a complete answer-cache design. See OAuth token introspection and caching.

Capture the observed result before manually invalidating state. Otherwise, a successful clean-up operation can conceal the failure of the normal revocation path. After remediation, repeat the warmed-state setup so the regression proves the changed control rather than merely an empty cache.

8. Resume delayed jobs and inspect conversation reuse

Owner: application engineer with conversation owner. Output: in-flight and historical-state decisions. Resume the synthetic job paused before result delivery. Check whether it uses a current policy decision at the approved output boundary. A retrieval-time check alone may not match the contract when authorization changes during generation.

Ask a follow-up in the warmed conversation. Inspect whether protected history is resubmitted as model context or returned directly to the reader. Define the permitted handling of prior messages explicitly. Content already received cannot be made unseen, but the application can still control future retrieval and redisclosure through its managed surfaces.

Check asynchronous exports and notifications if the application supports them. A queued report generated before revocation may be delivered afterward to a destination that no longer qualifies. Identify the delivery authorization boundary and test its actual decision, rather than assuming that a job accepted earlier remains valid indefinitely.

Record states that cannot currently be cancelled or rechecked. These are limitations requiring a policy and engineering decision, not exceptions to omit from the report. Restrict the affected feature if the observed behavior violates the accepted contract, then assign a remediation owner and retest plan.

9. Inject stale policy and dependency failures

Owner: security reviewer with identity engineer. Output: failure-mode evidence. In the isolated environment, make the authoritative policy source unavailable or deliberately delay its update. Observe whether the application serves a protected result from stale information. Define expected deny or bounded degraded behavior before running the fixture.

Do not prescribe one universal cache timeout. Choose the policy from the consequence of disclosure, the authoritative system's behavior and the application's operating constraints. If a product promises immediate enforcement, a test that accepts a delay contradicts that promise. Correct the implementation or the approved contract rather than redefining the test after seeing the result.

Test recovery too. Once the policy source returns, verify that the application observes the changed permission and does not resurrect stale results. A temporary denial can hide an old cache entry that becomes visible again later. Repeat protected and permitted control requests through the same client sessions.

Keep failure injection bounded and reversible. Record which dependency was affected and restore it through the supported procedure. If isolation fails or the exercise affects another tenant, use the stop control and treat the incident separately from routine fixture cleanup.

10. Adjudicate results against the declared boundary

Owner: independent observer with application owner. Output: disposition sheet. Classify each result as expected denial, permitted control, protected disclosure, unobservable behavior or unrelated service failure. Preserve the timing relationship to the authoritative change and the approved enforcement boundary. Do not combine all unsuccessful requests into one reassuring denial count.

| Tested surface | Required evidence | Failure disposition | | --- | --- | --- | | Existing-session search | Current policy and protected/control readback | Stop if protected snippet or answer returns | | Warm answer cache | Hit evidence and current permission disposition | Hold affected cache path on stale disclosure | | Source preview or download | Object-specific authorization observation | Restrict bypassing endpoint and retest | | Delayed job delivery | Start, change and output-boundary observations | Block obsolete output; preserve job identity | | Conversation follow-up | History reuse and response evidence | Restrict redisclosure until policy is enforced |

This sheet is a proposed acceptance artifact, not a report of tested production behavior. Add the actual environment, configuration revision, fixture identity, observation time and evidence reference to each row. State untested surfaces separately so an empty row cannot be mistaken for successful coverage.

If disclosure occurs, preserve restricted diagnostic evidence and stop the affected exercise path. Do not continue generating exposed outputs merely to improve the count. Assign the finding to the boundary that released the content, then investigate upstream contributors without treating a prompt change as an authorization fix.

11. Retest remediation and record acceptance limits

Owner: application owner with security reviewer. Output: signed technical acceptance record. Repeat the baseline, warm-state setup and failing case after the control changes. Include permitted readers and unrelated documents again. A fix that blocks everyone or deletes the fixture universally does not prove the intended per-reader boundary.

Acceptance criteria should name the tested permission path, identities, resources, states and enforcement timing. Record where the result is narrower than the product claim. An isolated test against one source connector cannot certify every enterprise integration, region or provider store.

For rollback, do not simply restore a version known to disclose protected data. A safer reversal may disable the affected feature while retaining the permission change. The service owner should approve the degraded experience and communicate what remains available. Security containment and ordinary deployment rollback are not always the same operation.

The next action is to add the demonstrated case to a release regression suite and identify which changes should trigger it: identity integration, search filtering, cache keys, conversation handling and asynchronous delivery. Keep fixture evidence and approved timing together so a future team does not weaken the assertion while updating the harness.

Use the AI deletion-path playbook when the requirement is removal rather than access change. For a broader engineering review, Ampity's AI systems work can start from these observed boundaries and acceptance limits. Reading and using this procedure does not require contact details.