The travel policy is thirty-seven pages long, and the employee has a specific question: will the company reimburse a hotel after a missed connection? An assistant trained on general information can produce a persuasive answer in seconds. The actual answer lives in the employer’s current policy, perhaps in a clause amended last month. A system using retrieval-augmented generation (RAG) searches for relevant source material and supplies it to a language model before the model responds. That sounds straightforward. In practice, the quality of the answer rests as heavily on the document search as on the model doing the writing.
A model’s fluency can disguise the fact that it has never seen the applicable document. Retrieval gives the application a chance to consult that document at the moment it needs it. The search can also fail, return obsolete material, or uncover information the employee has no permission to view. RAG creates an opportunity to ground a response in evidence; it does not guarantee that the evidence will be found or used correctly.
How RAG works, from question to answer
A common RAG workflow begins before anyone asks a question. An organization collects authorized documents, breaks long files into searchable passages, and makes them available through an index. Some systems represent passages as numerical embeddings so that searches can find related ideas expressed with different wording. Others use keyword search, and many combine approaches. An index might also store each passage’s title, publication date, and access rules.
When the employee asks about missed connections, the application searches for relevant passages. It may find a travel policy, a reimbursement schedule, and an outdated expense memo. The application then chooses which passages to place alongside the employee’s question in the model’s prompt. A language model generates a response conditioned on that retrieved text. A well-designed interface also shows source links, so the reader can check whether the explanation matches the policy.
Unlike fine-tuning, this process need not update the language model’s learned parameters. A newly indexed policy can become available to the assistant soon after a change. The timing depends on the ingestion process, however. A PDF replaced on a shared drive may still have an older copy in the search index. Good retrieval systems have to manage versions and removal as carefully as additions.
Why a RAG answer can still be wrong
Consider what happens if the search returns a policy from last year. The model may summarize it accurately, and the answer may still be out of date. Or retrieval may choose the wrong passage because the employee used an unfamiliar abbreviation. Even when the correct policy is found, the model might overlook an exception or combine two contradictory sources into a tidier rule than either actually states.
A retrieved document may also contain malicious instructions. Applications must treat source content as data to examine, not commands to obey. Permission filtering has to occur before a private document is supplied to a model, not after the answer has already been written. Citations help a reader investigate an answer, but their mere presence is not proof of accuracy.
The practical evaluation is end-to-end: did the search locate the right source; did the application respect access permissions; did the answer reflect the source; and did it admit when there was insufficient evidence? Testing only the model’s prose quality misses the chief reason RAG was added.
Does RAG require a vector database?
A vector database is one possible component of a RAG system, but many useful installations rely on conventional text indexes, relational databases, or a mixture of search methods. Size alone is a poor purchasing criterion. Document access permissions, update frequency, language coverage, latency, and the nature of the questions matter more than whether a vendor markets its product as AI-ready.
Our articles on vector databases and exact-word search explain the retrieval options in greater detail. For an expense assistant, the decisive test is less glamorous: when the travel policy has changed, can the system point an employee to the provision that actually governs the claim?
Sources and further reading
- Amazon Bedrock documentation: Knowledge Bases and RAG
- Amazon Bedrock documentation: Retrieval and response generation