Multi-Tenant RAG Architecture and Data Isolation

In multi-tenant RAG, retrieval is an authorization decision. A semantically relevant chunk is still forbidden if it belongs to another customer or a group the user cannot access.

Do not retrieve broadly and ask the model to ignore unauthorized context. Once the chunk enters the prompt, the isolation boundary has already failed.

Choose an isolation unit

Three patterns cover most systems:

A hybrid model is common: small tenants share infrastructure, while large or regulated tenants receive dedicated stores.

The choice should follow the threat model, tenant count, noisy-neighbor risk, deletion requirements, regional residency, and the search service's limits—not an assumption that vectors require one global index.

Derive scope from identity, never from the prompt

The authenticated application should produce an authorization scope:

{
  "tenant_id": "acme",
  "principal_id": "usr_42",
  "groups": ["support-emea"],
  "classification_max": "internal"
}

The retrieval API combines that scope with the user query. The model may propose search terms, but it cannot add tenants, groups, or classifications.

authenticated identity
        ↓
authorization service → permitted scope
        ↓
retrieval gateway(query, permitted scope)
        ↓
tenant-aware search store

Put the gateway in front of every vector, keyword, and document fetch path. Tool calls should not receive raw search credentials.

Shared-index rules

Every indexed unit needs immutable authorization metadata. At minimum:

{
  "tenant_id": "acme",
  "document_id": "policy-7",
  "allowed_group_ids": ["support-emea"],
  "classification": "internal",
  "source_acl_version": 12
}

Apply tenant and ACL constraints inside candidate retrieval. Post-filtering top-K results can damage both security assumptions and recall. If a vector engine retrieves 20 global neighbors and 19 belong to other tenants, filtering leaves one result—not the tenant's true top 20.

Test the filter behavior of the chosen engine under approximate search. The general threat and defense model in Securing AI Applications applies to the whole pipeline.

Scope every cache and durable object

Cross-tenant leaks often happen outside the vector index:

• semantic answer cache keyed only by question;
• reranking cache missing tenant or ACL version;
• conversation ID accepted without ownership validation;
• background task result fetched by guessable ID;
• trace viewer exposing raw retrieved text;
• document preview endpoint checking login but not document access.

Use an explicit scope in keys:

cache_key = tenant_id + principal_scope_hash + corpus_version + query_hash

When permissions change, the scope hash or ACL version must change too.

Keep an evidence-access audit trail

For sensitive deployments, record which chunk IDs were presented to the model, under which principal and policy decision. Avoid duplicating raw sensitive text when IDs and hashes are sufficient.

The trace design in LLM Observability: What to Log in Production should separate searchable security metadata from restricted prompt content.

An audit event should answer:

• who made the request;
• which tenant and groups were active;
• which policy version allowed access;
• which documents entered the final context;
• which service account performed the read.

Build adversarial isolation tests

Create two tenants with deliberately similar documents:

Acme cancellation code: ACME-481
Beacon cancellation code: BCN-992

Then test:

• direct requests for the other tenant's code;
• paraphrases and misspellings;
• copied text from the other tenant's document;
• conversation switches between tenant sessions;
• cache warm-up by one tenant followed by the same query from another;
• ACL removal while caches and long-running tasks are active;
• document deletion and re-indexing.

Assert on retrieved chunk IDs, not only the final answer. A model refusal can hide unauthorized retrieval.

Isolation readiness checklist

• The isolation pattern matches regulatory and operational requirements.
• Tenant scope comes from authenticated state.
• Every retrieval path passes through one authorization gateway.
• Filters run during candidate retrieval.
• Cache, conversation, job, trace, and preview access are tenant-scoped.
• Permission changes invalidate derived artifacts.
• Audit logs identify every context document and policy decision.
• Cross-tenant tests assert that forbidden chunks were never retrieved.

Sources

• Microsoft: Secure multitenant RAG inferencing architecture
• Microsoft: Multitenancy and content isolation in Azure AI Search
• Microsoft: Azure AI Search security best practices