RAG Freshness, Updates, and Document Deletion

RAG freshness is a data-lifecycle contract, not a scheduled re-embedding job. Give every source a stable identity, checksum, version, authorization scope, and lifecycle state. Build new chunks under a new version, activate them atomically, tombstone deletions before asynchronous cleanup, invalidate derived caches, and verify that stale evidence is no longer retrievable.

Deleting the source file alone is not deletion. Copies may remain in parsed artifacts, vectors, keyword indexes, reranker caches, response caches, traces, and evaluation datasets.

Use a source registry as the control plane

Keep an authoritative record outside the vector index:

The production RAG architecture places this registry in the knowledge plane. The vector database is a serving index, not the only ledger of truth.

Model updates as version transitions

Do not overwrite chunks one by one while users query the same version. A safer sequence is:

observed → fetched → parsed → validated → indexed → activated
                ↘ quarantined
active old version → superseded → deleted after retention policy

Build the new version with deterministic chunk IDs such as:

chunk_id = hash(scope_id, source_id, source_version, chunk_locator)

Validate completeness and authorization metadata before activation. Then switch the active version pointer. Queries must filter to lifecycle_state=active and the authorized scope, so partial new versions and superseded chunks cannot leak into results.

If the new version fails validation, keep the last known good version active and alert on freshness lag. Silent partial activation is worse than a visible stale state.

Deletion is a distributed transaction

Use a tombstone-first workflow:

1. Authorize and record the deletion request with scope_id and source_id.
2. Mark the source tombstoned in the control plane.
3. Make the serving path exclude tombstoned content immediately.
4. Enumerate derived artifacts from the chunk manifest.
5. Delete vectors, keyword documents, parsed objects, caches, and other governed copies.
6. Verify absence using IDs and retrieval probes.
7. Store per-system receipts and mark deletion complete.

Tombstoning closes the user-visible access path while eventually consistent stores finish physical deletion. It is not a substitute for deletion where policy requires erasure.

Pinecone documents deletion by record ID, metadata filter, namespace, and index, and notes that data visibility is eventually consistent. Qdrant supports point IDs, payload filters, and explicit delete operations. Those APIs are execution mechanisms; your registry still needs to know exactly which records belong to a source.

Prefer deletion by manifest IDs. Broad metadata filters are useful for recovery but can over-delete if metadata is incomplete or scope is omitted.

Handle every derived copy

Create a deletion coverage matrix:

Do not promise that backups, security logs, or legally retained records are erased through the serving-index job unless they actually are. Document their separate policy and access controls.

Measure freshness as lag, not vibes

Track at least:

observation_lag = observed_at - source_updated_at
processing_lag  = activated_at - observed_at
freshness_age   = now - source_updated_at(active_version)
deletion_lag    = deletion_completed_at - deletion_requested_at

Segment by connector, tenant, source family, and lifecycle state. Alert on the oldest active version and tombstones past their deletion objective, not only average processing time.

The source timestamp may be missing or unreliable. Preserve both source-reported and system-observed times instead of silently treating ingestion time as document time.

Test plan

Add retrieval cases for questions answerable only by the removed version. A direct ID lookup can be empty while a cached response still cites the deleted text.

Use the RAG failure decision tree when an old answer appears, then measure retrieval and final behavior separately with the RAG evaluation scorecard.

Production checklist

• Stable source identity survives path and title changes.
• New versions activate only after completeness and policy checks.
• Queries require active lifecycle state and authorized scope.
• Tombstones block serving before asynchronous cleanup.
• A manifest enumerates every derived chunk and artifact.
• Cache keys include source/index versions.
• Deletion produces per-system receipts and absence checks.
• Freshness and deletion lag have objectives and owners.
• Update, partial failure, retry, and delete races are tested.

This lifecycle is what makes “current knowledge” an engineering claim rather than marketing copy. JoinAI’s AI Engineer MasterClass covers the production RAG and evaluation practices behind it.

Sources and further reading

• Pinecone: Delete records and data freshness
• Qdrant: Point storage and modification
• Qdrant: Payload filtering