What is retrieval-augmented generation (RAG)?
Retrieval-augmented generation (RAG) is a technique that pairs a search step with a large language model. When a user asks a question, the system first retrieves the most relevant passages from a store of the company’s documents, then hands those passages to the model with instructions to answer from them. The model writes the answer and cites the passages it used.
RAG exists because a language model knows nothing about a specific company and cannot be kept current by retraining. Retrieval supplies the facts at the moment of the request. The model supplies the reading comprehension and the writing.
How retrieval-augmented generation works
RAG has two pipelines. The ingestion pipeline runs ahead of time. Documents are collected from file shares, wikis, ticketing systems, or a document management system. Each document is split into chunks of a few hundred words. An embedding model turns each chunk into a vector that captures its meaning, and the vectors are stored in a vector database together with the chunk text and metadata such as source, date, and the groups allowed to see it.
The query pipeline runs on each request. The question is embedded with the same model, and vector search returns the chunks whose meaning is closest. Many systems combine this with keyword search (hybrid search) so that exact part numbers and names are not missed, then apply reranking to put the best few chunks first. Those chunks, the question, and a system prompt go to the model, which produces a grounded answer with citations.
Retrieval-augmented generation in practice
A building-products distributor has forty years of manufacturer spec sheets, installation guides, and internal bulletins spread across a file server. Inside sales representatives spent minutes per call hunting for load ratings and compatibility notes. A RAG assistant indexes the documents nightly, and representatives ask questions in plain language while the customer is on the line. Each answer shows the source page, so the representative can confirm it before quoting. When a manufacturer issues a revised sheet, the next nightly run replaces the old chunks and the answers change with it.
In Kirk Tech Solutions engagements, RAG over an existing document store is usually the first generative AI deployment, because it requires no model training, shows its sources, and respects the access permissions the documents already carry.
Retrieval-augmented generation compared with fine-tuning
| Aspect | RAG | Fine-tuning |
|---|---|---|
| What it adds | Facts from documents at query time | Style, format, and vocabulary built into the weights |
| Keeping current | Re-index the changed documents | Retrain and redeploy the model |
| Traceability | Every answer can cite its sources | No way to trace an answer to a training example |
| Access control | Filter retrieval by the user’s permissions | Everything in the training set is available to every user |
| Data needed | The documents as they exist | Curated examples of inputs and ideal outputs |
What to watch for
- Retrieval quality decides answer quality, so test the search step on its own with real questions before blaming the model.
- Apply permission-aware retrieval from the start, because an index that ignores document permissions exposes HR and finance files to anyone who asks.
- Plan the refresh schedule and the handling of deleted or superseded documents, since stale chunks produce confident, outdated answers.
- Expect scanned PDFs, spreadsheets, and tables to need special handling, because naive text extraction destroys their structure.
- Instruct the model to decline when retrieved passages do not answer the question, and measure how often it does so correctly.