Hybrid DNS: Find the Effective Rule Before Changing the Forwarder

Trace an exact DNS question through VPC and Profile associations, private zones and conditional forwarders. Separate missing records, wrong associations and...

Primary sources checked 2026-10-09

Identify the question and its effective DNS configuration

When an AWS workload and an on-premises workload resolve the same internal name differently, start with the exact DNS question and the resolver context that receives it. Find the applicable private hosted zone or forwarding rule, including its association path. Then investigate the selected path. Changing the nearest visible forwarder can move unrelated names or create another dependency without correcting the original failure.

The output of this guide is a name-resolution decision record: what should handle one question, what evidence supports that expectation, what was observed, and who owns the next bounded action. A documented selection rule can support an expectation. Only a query from the intended client context can supply that client's observed response, and a DNS response still does not establish application access.

The scenario uses Route 53 VPC Resolver, default inbound endpoints and conditional forwarding for a fictional internal namespace. It also examines simple local-versus-Profile selection cases. It excludes Route 53 Global Resolver, delegated inbound endpoints, DNSSEC diagnosis, reverse DNS, wildcard and alias evaluation, and a complete implementation of every Resolver rule interaction. Those exclusions require separate analysis when present; they are not permission to ignore a relevant setting.

All addresses, inventories and outcomes below are stipulated examples. The accompanying offline fixture checks a deliberately limited decision model. No AWS API, DNS query, customer network, packet capture or application request was executed. No DNS configuration was changed. These figures explain stipulated configurations, not observed DNS or application behavior.

Keep resolution authority, forwarding and application access separate

A private hosted zone contains records for a namespace. A conditional forwarding rule selects another resolver to ask. The forwarded-to server might be authoritative for that name, or it might forward again. Merely finding a target address does not establish where the answer originates. Record the zone owner separately from each forwarding operator.

An inbound endpoint provides an entry to VPC Resolver. An outbound endpoint provides the path used by forwarding rules toward target resolvers. AWS describes the outbound endpoint's VPC separately from the VPCs associated with a rule. Hosting the endpoint in a hub does not, by itself, associate its rules with every application VPC. AWS outbound flow.

The client can subsequently connect to an address returned in the answer through a different network path. A DNS query reaching a hub endpoint is not an application request traversing that endpoint. The S3 caller-path article works through that distinction for an S3-specific endpoint decision. Do not generalize its private-DNS arrangement to every internal service.

Write the question as an absolute name and a type, such as orders.internal.example.com. and A. Record the process that issued it, not just the engineer's location. Search suffixes, an application proxy, a container resolver or a local cache can make a convenient command exercise a different question. Preserve the question seen by the selected resolver when evidence permits; mark the relationship unknown when it does not.

Collect both direct and Profile association paths

For a VPC-origin query, start with that VPC's identity, account, Region and resolver configuration. Collect direct private-zone associations and Resolver-rule associations. For a remote query entering through a default inbound endpoint, identify that endpoint and its VPC context. The inbound path does not inherit a private zone merely because some other VPC contains the application the caller hopes to reach. AWS inbound flow.

Use existing authorized read access. An access denial means the collector lacks evidence; it does not mean the association is absent. Keep pagination, status and collection time with the inventory. A half-collected list cannot support a negative statement about a more-specific rule.

Profiles need a second association chain. Establish which Profile applies to the VPC and which DNS resources belong to that Profile. AWS explicitly warns that ListHostedZonesByVPC does not include associations through Profiles. An empty result from that API alone cannot establish that a VPC has no private namespace. Hosted-zone listing boundary.

The Profile-to-VPC relationship and resource-to-Profile relationship are separate records. Their list APIs return association identities and status; retain both paths and complete their pagination. A resource in a Profile that is not associated with the caller's VPC is not evidence of applicability there. Profile associations, Profile resource associations.

The worksheet also needs relevant DNS Firewall and resolver settings. Store restricted identifiers in the approved evidence location and use aliases in shared discussion. This guide supplies no broad IAM policy, collection script or instruction to grant yourself access across accounts.

Apply precedence only to a complete, applicable candidate set

AWS matches forwarding rules against an exact domain or its subdomains and selects the most-specific matching rule. A label boundary matters: orders.internal.example.com. matches internal.example.com., but notinternal.example.com. does not. The rule must apply to the VPC context under review. Rule matching.

For overlapping private zones associated with the same VPC, the most-specific zone is selected. A missing name in that selected zone does not cause a search through a less-specific private zone or a public fallback. For an associated private zone and an associated forwarding rule with the same domain, the forwarding rule takes precedence. Private-zone considerations.

Treat these as bounded rules, not a shortcut that “forwarding always wins.” Compare actual suffixes, resource kinds and association paths. The offline model deliberately holds mixed zone/rule candidates with differing suffixes instead of claiming to reproduce the entire service. It also holds a mixed resource-kind tie across local and Profile paths until the exact combination has been resolved with adequate documentation and scoped evidence.

AWS's Profile examples show a local VPC rule winning an equal-name conflict and a more-specific Profile rule winning over a broader local rule. Consequently, “local always wins” is also unsafe. Keep equal-name and more-specific comparisons separate, and identify which resources the comparison involves. Profile prioritization.

System rules can exclude a subdomain from a broader forwarding rule so VPC Resolver handles it. A system rule does not create the required record. Delegation rules have a different activation mechanism involving matching delegation information and NS responses; they are not conditional forwarders with another label. Inventory their presence and stop the fixture's simplified selection when they are relevant. Rule types, delegation distinction.

Work the name from three origins before proposing a correction

The fictional inventory contains application VPC app-a, hub VPC hub-h, on-premises resolver site-r, and a default inbound endpoint in hub-h. A separate on-premises resolver, site-edge, supplies the remote entry path by forwarding the example suffix to that inbound endpoint. All are in a stipulated compatible arrangement; no account or Region has been inspected. The private zone internal.example.com. is directly associated with both VPCs and contains orders.internal.example.com. A 10.20.0.20. The same-domain conditional rule forward-internal targets site-r, where the stipulated authoritative answer is 10.50.0.20.

These different answers are a diagnostic input, not an assertion that split-view DNS is inherently broken. The application and DNS owners must specify the intended answer set for each origin. If they cannot agree which system owns the name, a network change would implement an unresolved product decision.

Case A: Same-suffix forwarding overrides the local private zone

Associate forward-internal with app-a in the fictional inventory. An uncached orders.internal.example.com. A question through app-a's Resolver selects that forwarding rule. The expected target is site-r; the private-zone record does not win this equal-domain case. The target's stipulated answer remains hypothetical, and the fixture reports a selected forward target rather than inventing a received response.

If the owner intended the AWS address, first ask whether the whole suffix belongs in AWS. Removing the rule could change every covered name without a more-specific selection. A correction needs an inventory of those names, their owners and representative negative cases, not only the one failing application.

Case B: A rule associated only with the hub does not explain the app query

Change only the association: forward-internal applies to hub-h, not app-a. The app's same question now selects its associated private zone in this bounded inventory. A screenshot of the rule and healthy outbound endpoint would miss the association error. The expected private answer is 10.20.0.20; actual DNS response and application behavior are still unobserved.

For a remote request that site-edge forwards to the hub's default inbound endpoint, investigate hub-h separately. With the hub rule present, the same question can take the forwarding path to site-r there. Two origins need two records even when the displayed QNAME is identical. Do not “fix” the app association by adding it until the name owner confirms that the remote answer is also right for the app.

Case B keeps site-edge and site-r separate. The private zone applies to app-a; the same-suffix forwarding rule applies only to hub-h and wins there. Blue arrows mean DNS questions, thin gray arrows show expected answer direction, and dotted purple lines mean association rather than traffic. The mobile composition omits return arrows for space; it does not imply one-way DNS. All paths and expected addresses are fictional. No response, transport or application operation was observed.

Case B app origin only. The private-zone association is arrowless; blue shows the question and gray the expected answer direction. Hub-only forward-internal does not apply to app-a. DNS response and application access remain unobserved.

Case B remote origin, under complete direct associations and default inbound premises. The same-domain hub forwarding rule wins there. site-edge is the entry role; site-r is the distinct terminal authority, not a return forwarder. The expected address is documentary, not received.

Case C: The child zone contains no such name

Use a separate fixture with no matching forwarding rule. Associate both internal.example.com. and dev.internal.example.com. with app-a. Put orders.dev.internal.example.com. A 10.20.0.30 only in the parent zone and stipulate that the child contains no name orders.dev.internal.example.com.. The child wins; the expected result for this nonexistent-name case is NXDOMAIN, not the parent's address.

That conclusion requires a complete child-zone record inventory and no relevant wildcard, alias or other excluded behavior. If the name exists but the requested type is absent, this fixture holds the response interpretation instead of generalizing nonexistent-name handling to every empty-answer case. Record the real response code and answer section independently.

Case D: A narrower rule changes the target

Associate forwarding rules for internal.example.com. and dev.internal.example.com. with app-a, without overlapping private-zone candidates in this fixture. The question orders.dev.internal.example.com. A selects the child rule. A successful test of orders.internal.example.com. establishes nothing about that child target. Test a representative name on each side of the rule boundary and keep the expected target explicit.

Cases E and F: Equal-name and narrower Profile rules differ

For the equal-name comparison, stipulate a complete Profile path to app-a and two forwarding rules for internal.example.com., one local and one through that Profile. The local rule wins. For the narrower comparison, change the Profile rule to dev.internal.example.com. and ask orders.dev.internal.example.com. A; the Profile rule wins. Both fixtures involve forwarding rules, not a speculative mixed zone/forwarding tie.

If only the Profile resource association is known, neither case is established. The Profile-to-VPC link must also be known and complete. The negative fixture removes that evidence and requires HOLD, rather than quietly falling back to the local rule.

Demonstrate a configured cycle without diagnosing every timeout as one

Now stipulate that site-r conditionally forwards internal.example.com. to the hub inbound endpoint, while the hub's matching rule forwards the same question back to site-r. A trace of this configured path revisits the same resolver context for the same question. AWS warns against associating a rule with the VPC of an inbound endpoint reached directly or through an on-premises server because this arrangement can loop. Avoiding loop configurations.

The offline trace identifies a configured cycle and holds the proposed path. It does not report an observed loop, an exact retry count or a timeout duration. A runtime trace might be interrupted earlier by filtering, unavailable transport, cached data or another condition. A timeout without the forwarding configuration leaves the cause unresolved.

Only this changed fixture makes site-r forward back to the hub's default inbound endpoint. The closed blue path revisits the same hub-h context for the same question and applicable rule; it is configuration evidence, not a captured runtime loop. The timeout-only record supplies no such path and stays UNKNOWN. Conflicting or incomplete associations also remain HOLD. Neither view establishes execution, retries, timing or an authoritative answer.

Changed fixture only: site-r now forwards back to the same hub context for this exact question and applicable rule. The closed blue path supplies configuration-cycle evidence. No answer, executed loop, retry count or timeout duration is established; HOLD the path change.

Separate timeout-only evidence does not prove the preceding configured cycle. Selected rule, target behavior and terminal authority remain UNKNOWN. Gather question-specific path evidence; no DNS execution or application result is supplied.

Trace the entire selected path, including the on-premises resolver's effective conditional rules and terminal authority. Do not draw every configured target as if it was visited in one query. AWS says outbound target selection is random, with retry to a random target after no response. Target-list order is not primary/secondary priority. Each possible target needs consistent namespace intent and assessed reachability. The fixture holds branching paths it does not model. Outbound target behavior.

The bounded correction might remove an unintended hub association while retaining the required spoke association, or change the authoritative namespace boundary. It is not automatically “delete one arrow.” Identify the clients relying on that relationship and the authority that should terminate the question before proposing a change. Preserve their negative and positive name tests for review.

Distinguish policy, transport, cache and service observations

For standard DNS in this scenario, AWS documents inbound TCP and UDP port 53 access and corresponding outbound DNS access toward the network. Inspect the exact endpoint security groups, routes, network ACLs and remote firewall policy under the network owner's authority. A permitted UDP test does not establish the TCP path, and an endpoint association cannot establish either transport. Custom ports or protocols require their own documented configuration. Inbound endpoint setup, outbound endpoint setup.

DNS Firewall can affect queries from a VPC and queries entering that VPC through Resolver endpoints. Its rules, actions and failure-mode configuration belong in the evidence record. A block or modified DNS response must not be misdiagnosed as proof that a private-zone record is missing. This guide does not authorize disabling filtering or changing failure mode for convenience. DNS Firewall behavior.

Compare a warm application process with a fresh process from the same intended context. A fresh process is not proof of an uncached recursive lookup. AWS Resolver query logging omits queries answered from its cache, so an absent new log entry cannot by itself prove that no DNS query occurred. Preserve log scope, collection interval, query name/type and response evidence. Query-log limitations.

Use existing approved telemetry first. Creating query logs or packet captures can change cost and expose internal naming data; those actions need separate authorization and retention controls. If a diagnostic needs an uncached query, agree a safe test name and observation method with the zone owner. Do not generate arbitrary production names or flush shared resolver caches as an unreviewed experiment.

The DNS failover article owns cache age, runtime lookups and connection reuse. Carry its distinction into the final acceptance test: record DNS result, newly established connection and authenticated application operation separately. None of them alone substitutes for the others.

Choose an alternative at the namespace boundary

Retain on-premises authority when that owner must continue maintaining the name and the forwarding dependency is acceptable. The decision requires usable paths from each origin, consistent target behavior and an operating owner for network interruptions. It does not become preferable merely because the old record already exists.

Use a private hosted zone as the intended authority when the responsible team can maintain the required records and associate the relevant VPC contexts. Review the fate of covered remote clients before narrowing or removing a forwarder. A missing child record is a record-ownership problem; adding an unrelated public record would not repair the selected private namespace.

Split a child namespace when different owners genuinely need separate authority. For example, a deliberate aws.internal.example.com. boundary can reduce overlap if consumers and owners accept the names. Renaming can affect certificates, application configuration and retained references, so it is not a cost-free DNS-only fix. A system-rule exception may be another candidate under a broader forwarder; validate the resulting resolver behavior and records before choosing it.

Profiles can make distribution consistent across VPCs, but they do not eliminate the need to inspect effective local conflicts. Delegation may fit a genuinely delegated namespace; it needs an NS/delegation design rather than copying the conditional-forwarding fixture. Leave the decision held when the candidate's scope or evidence falls outside this guide.

Copy the complete blank resolution worksheet

Use one record for one QNAME/QTYPE and origin, with linked records for related cases. Every blank means evidence still required. Keep configuration expectation and observed response in different fields. The fields are grouped for reading on a narrow screen, not compressed into a wide operations table.

1. Decision and authorityRecord ID, time, DNS owner, application owner, authorized collector and approved observation scope: not supplied.

2. Exact questionAbsolute QNAME, QTYPE and intended answer or negative result for this origin: not supplied.

3. Query originProcess/runtime, VPC or site, account, Region and relevant subnet or network context: not supplied.

4. Resolver entryActual configured resolver; default inbound endpoint and its VPC if used; evidence tying the process to that path: not supplied.

5. Direct associationsPrivate zones, forwarding/system/delegation rules, statuses, resource identities, complete pagination and dated references: not supplied.

6. Profile pathProfile-to-VPC and resource-to-Profile links, statuses, complete pagination, or evidenced absence of a Profile: not supplied.

7. Relevant controlsVPC DNS settings, DNS Firewall, DNSSEC and other behavior that could invalidate the simplified model: not supplied.

8. Selection explanationMatching suffixes, resource kinds, association paths, selected candidate and documentation or unresolved conflict: not supplied.

9. Terminal authoritySelected zone/record inventory or every possible forward target and subsequent owner; missing or ambiguous hop: not supplied.

10. Loop assessmentQuestion-specific configured path, revisited context if any, configuration references and runtime evidence kept separate: not supplied.

11. Transport evidenceEndpoint paths, source/destination context, approved TCP/UDP checks, network controls and unknowns: not supplied.

12. Observed responseTime, client context, exact query, resolver, response code, answer, TTL/cache limitations and evidence reference: NOT EXECUTED.

13. Application evidenceWarm/fresh client, new/reused connection and authorized operation result: NOT EXECUTED.

14. Proposed actionNo change, record correction, association correction, namespace redesign or HOLD; affected names/origins and owner: not supplied.

15. Stop and restorationPre-change revision, authorized reversal, lost evidence or unexpected response stop condition, cache/client recheck: not supplied.

16. Closure and expiryAccepted bounded evidence, unresolved dependencies, approver and changes that invalidate this record: not supplied.

Read a filled record without mistaking it for test evidence

This filled version covers Case B from the app origin. Every configuration item is a fictional premise. Its purpose is to show a useful held handoff even when the selection expectation is clear. It contains no fake console output or query transcript.

1. Decision and authorityI12-B, illustrative revision 1. DNS and application owners are role labels, not named reviewers. Scope is offline reasoning only.

2. Exact questionorders.internal.example.com., A. Application owner stipulates the intended AWS answer 10.20.0.20.

3. Query originFictional orders worker in app-a, symbolic account A and Region R. No real subnet or runtime inspected.

4. Resolver entryPremise: the worker uses app-a's VPC Resolver. It does not use hub-h's inbound endpoint for this case.

5. Direct associationsComplete synthetic inventory: zone internal.example.com. applies to app-a and hub-h; forward-internal applies only to hub-h.

6. Profile pathPremise: no Profile applies to app-a. This is explicit fixture data, not an inference from an empty direct-zone listing.

7. Relevant controlsPremise: required VPC DNS settings support private resolution; no relevant DNS Firewall block or excluded rule/record behavior. Real control evidence is absent.

8. Selection explanationThe hub-only forwarder is inapplicable to this app-origin question. The associated private zone is the candidate under the remaining complete premises.

9. Terminal authorityStipulated simple record orders.internal.example.com. A 10.20.0.20 in that zone. No downstream DNS forwarding in this fixture branch.

10. Loop assessmentNo forwarding trace claimed for the app's private-zone branch. The hub/remote cycle case is a separate record and cannot be transferred to this origin.

11. Transport evidenceUNKNOWN. No endpoint or client transport was exercised. Healthy endpoint status was not used as a substitute.

12. Observed responseNOT EXECUTED. Expected private answer only; no observed response code, TTL or query-log entry.

13. Application evidenceNOT EXECUTED. Address selection, TLS identity and authenticated orders read remain outside the fixture.

14. Proposed actionHOLD any association change. Obtain the actual app resolver path and scoped query evidence. A hub screenshot is insufficient reason to attach the rule to app-a.

15. Stop and restorationNo change made. A later authorized change must preserve the prior association revision and stop on an unexpected answer for either the selected or unaffected control names.

16. Closure and expiryOnly the offline selection expectation is established. Operational acceptance remains open; association, Profile, record, client or filtering changes require a new review.

Use offline checks to challenge reasoning, then plan a bounded observation

The offline fixture tests selection and configuration tracing with expectations declared separately from its implementation. Its negative cases include a hub-only association, an incomplete Profile chain, a missing child-zone name, a label-suffix false match, conflicting same-specificity candidates, an unsupported rule kind, and timeout-only evidence. A green fixture run means those assertions behaved as expected locally. It cannot validate AWS permissions, current deployed settings, propagation, reachability or any real response.

Download the offline hybrid DNS authority fixture (ZIP). Extract all four files into one directory and use Node.js 22 or later. Run node --test resolution-model.test.mjs there. The standalone package has 38 local tests: 27 selection cases, eight configured-path cases and three safety checks. Its README describes the deliberately unmodelled boundaries. The package contains no AWS client, credentials or network collection code.

For an authorized observation, the DNS owner first approves the exact names, client contexts and collection method. Establish a baseline and preserve the configuration revision. Observe the selected question from each required origin, retain status and answers, and compare them with the worksheet. Where cache state is unknown, keep that limitation in the result rather than claiming a fresh recursive lookup. Include an unaffected name and a deliberate negative name agreed by its owner.

A later change requires its own approval, affected-population review and restoration procedure. Prefer a bounded association or record change only after the worksheet identifies why it is needed. Stop when the observed resolver context differs, an unexpected name changes, a required origin loses service or evidence becomes incomplete. Preserve the observed result before another change obscures the cause.

Restoring the prior configuration does not erase cached responses or prove application recovery. Recheck the same origins and agreed operations after reversal, account for the cache behavior observed in that environment, and record any client intervention. Never disable hostname verification, redirect every domain or remove a security control to make one lookup succeed.

The handoff is complete when the owners can explain the selected authority and association path, distinguish expected from observed outcomes, and assign every unresolved field. If those owners need help designing the bounded test, an Ampity reliability review can scope the evidence and acceptance criteria. Take one exact question and its blank worksheet to that review; do not arrive with a preselected forwarder change.