A RAG system can derive an answer correctly from a document and still be wrong because the document is outdated. Retrieval quality therefore begins before embeddings and vector search: with source authority, modification data, and a controlled update process.
Freshness is not a single timestamp. It is the verifiable chain from the source system to the answer.
Separate authority from recency
The newest document is not automatically authoritative. Yesterday’s chat transcript may contradict an approved policy from last week. For each content type, define which source takes precedence and who approves it.
Store authority level, owner, validity date, and version as metadata. Retrieval should account for these signals rather than ranking only by semantic similarity.
Measure the update path
For each source, the following should be visible:
- when it was changed in the source system,
- when the connector detected the change,
- when processing and indexing were completed,
- which version is currently retrievable,
- whether there is an error or backlog.
The time between a source change and the availability of the retrievable version is the true update latency.
Prioritise deletions and permission changes
An update matters. A deletion or revoked permission is often more urgent. The index must reliably reflect removed content and access changes.
Plan dedicated events and tests for these cases. A periodic full reconciliation can act as a safety net, but it does not necessarily replace rapid revocation for sensitive data.
Tie caches to versions
Response and retrieval caches can continue serving stale content even after the index has been updated. Tie cache keys to the source version, policy version, and permission context. Define the events that trigger invalidation.
Caching may be entirely unsuitable for current prices, availability, or requirements.
Make stale sources visible
A system should not silently pretend that every source is current. Display the document date, version, and, where appropriate, a warning. Critical workflows can enforce a maximum permitted source age.
If no source falls within that limit, the agent should stop or request human review.
Evaluate freshness
Test changes under controlled conditions: a new version, a conflicting version, a deletion, and a permission revocation. Measure how quickly the system reflects the new state and whether old answers still emerge from caches.
These tests belong in ongoing operations, not only in the initial implementation.
Sources
- NIST AI Risk Management Framework: https://www.nist.gov/itl/ai-risk-management-framework
- Google Cloud, RAG Engine: https://cloud.google.com/vertex-ai/generative-ai/docs/rag-overview
- Microsoft, Advanced RAG guidance: https://learn.microsoft.com/en-us/azure/developer/ai/advanced-retrieval-augmented-generation
- OWASP Top 10 for LLM Applications: https://owasp.org/www-project-top-10-for-large-language-model-applications/
Next step
Choose three authoritative sources and perform one update, one deletion, and one permission revocation for each. Measure the time until retrieval is correct, and also check every response and gateway cache.