In brief
- Retrieval-augmented generation (RAG) supplies passages from your documents at question time. Fine-tuning changes the model's weights using examples to improve a measured task or behaviour.
- Retrieval is a useful starting point when answers need current company knowledge, source references and permission checks. Those features need to be built and tested. Fine-tuning can help with a stable format or repetitive task.
- Whichever you choose, the deciding work is retrieval quality, access control and evaluation against real questions, including the ones the documents cannot answer.
Teams that want an assistant over their own documents, tickets and data usually ask the same first question: should we fine-tune a model on our content, or use retrieval? The short answer is that these techniques solve different problems, and for internal knowledge the answer is usually retrieval first, with fine-tuning added later only if a specific, measurable need appears. Here is the reasoning, without the jargon.
What each approach actually does
Retrieval-augmented generation
RAG splits your documents into passages, indexes them (typically with embeddings and often a keyword index as well), and at question time retrieves the passages most relevant to the question. Those passages are placed in the model's context along with the question, and the model writes an answer grounded in them. The model itself is unchanged; what changes is what it is shown.
Fine-tuning
Fine-tuning continues training a model on examples of inputs and desired outputs, adjusting its weights. It can teach a tone of voice, an output format, domain vocabulary or a narrow classification task. Facts learned in weights are harder to keep current, and those weights do not provide a reliable source reference. A fine-tuned model can still use retrieval when it needs current, attributable information.
A side-by-side comparison
| Consideration | Retrieval-augmented generation | Fine-tuning |
|---|---|---|
| Keeping up with changing content | Refresh the index, then check that updated passages are retrieved; timing depends on the ingestion schedule | Updating facts learned in weights requires another training and deployment cycle; a fine-tuned model can also use retrieval |
| Citing sources | Retrieved passages can support citations, which still need to be checked against the answer | Fine-tuning alone does not provide source provenance; retrieval or another evidence layer can add it |
| Respecting user permissions | Enforce source permissions before retrieval and test isolation between users | Model weights do not enforce document permissions; keep restricted knowledge in an access-controlled retrieval layer |
| Handling questions outside the material | Test retrieval confidence and a refusal path; retrieval alone does not prevent unsupported answers | Train and evaluate refusal behaviour; fine-tuning alone does not establish that an answer is supported |
| Teaching format, tone or a narrow task | Possible through prompting, with limits | Strong: this is what fine-tuning is for |
| Upfront cost | Engineering time for ingestion, retrieval and evaluation | Preparing representative training and evaluation examples, plus training runs; the amount depends on the task and model |
| Ongoing cost | Longer prompts per query; index maintenance | Hosting or serving a custom model; periodic retraining |
When RAG is the right choice
- The assistant must answer from documents, policies, tickets or records that change over time.
- Users need to see where an answer came from before they trust it.
- Different users are allowed to see different things, so permissions must be enforced at retrieval time.
- The content is too large or too varied to summarise into training examples.
These requirements arise in HR and policy questions, engineering runbooks, customer-support knowledge, and contract and procedure lookups.
When fine-tuning earns its place
- A narrow, repetitive task with a stable format: classifying tickets, extracting fields from a known document type, writing in a strict house style.
- Prompting has been tried and measured, and the remaining error rate is about behaviour rather than missing information.
- Latency or cost per request matters enough that a smaller, specialised model is worth the maintenance.
Fine-tuning and RAG are not mutually exclusive. A common mature design uses retrieval for knowledge and a lightly fine-tuned or carefully prompted model for consistent behaviour. Start with retrieval and good prompting; add fine-tuning when you can name the metric it will improve.
The work that actually decides success
Retrieval quality
Check whether the right passage reached the model before changing the model itself. Document structure, semantic and keyword search, metadata filters and re-ranking can all affect retrieval. Measure retrieval separately from answer quality so you can tell whether the source was missing, the wrong passage was selected or the answer misused the evidence.
Access control
Permissions must be enforced in the application before retrieval, using the user's identity and the source system's rules. Instructions written into a prompt are not access control; a determined user can talk a model out of them. This is a design requirement, not a feature to add later.
Evaluation with real questions
Collect the questions people actually ask, including ones the documents cannot answer, and review the assistant's behaviour on them before widening the audience. Score groundedness (is the answer supported by the retrieved passages?), correctness, and how the assistant behaves when it should decline. Keep the set and re-run it after every change to the index, prompt or model.
Unanswerable questions
A good assistant says "I could not find this in the approved sources" and points to a human. A bad one improvises. Build the decline path deliberately, test it, and make it easy for users to report wrong answers.
A sensible first project
Choose one defined collection of documents and a small group of users. Collect twenty to fifty real questions. Build retrieval and answering over that collection, with source references and a decline path, and review the results with the users. Only then decide on wider access, automated actions or any fine-tuning. Agree the timeline after checking source access, permissions and the evaluation requirements.
For the underlying approach, see the original RAG research paper and AWS's guide to retrieval and source citations. Implementation choices still need testing against your own sources and users.
Frequently asked questions
Does fine-tuning make a model "know" our documents?
A model may learn information during fine-tuning, but reliable recall, updates and source citations are separate requirements. For answers about current documents, retrieval supplies the passages that can support the answer. Fine-tuning can be combined with that retrieval when a measured behaviour or format requirement justifies it.
Will our data be used to train someone else's model?
It depends on the provider and the contract terms you select. Business and API tiers of the major model providers generally state that customer data is not used for training, but this should be confirmed in writing and reflected in your architecture, including where indexes and logs are stored.
How much does an internal knowledge assistant cost to run?
Ongoing costs come from model usage per query, the retrieval infrastructure, and keeping the index current as documents change. For most internal tools these are modest compared with the engineering time to build and evaluate the system properly. Ask for both numbers when comparing proposals.
Scope a knowledge assistant with MV.tech
Our AI consulting service covers source preparation, retrieval, access rules and evaluation. If the assistant also needs live systems, review MCP servers and AI connectors. Bring one source collection and a few real questions to a 30-minute scoping call.