In brief
- Retrieval-augmented generation (RAG) gives a language model the relevant passages from your own documents at question time. Fine-tuning changes the model's weights using examples, which teaches behaviour and style far better than it teaches facts.
- For assistants that answer questions about company knowledge, RAG is almost always the right starting point: it stays current, can cite sources and can respect user permissions. Fine-tuning suits narrow, repetitive tasks with a stable format.
- Whichever you choose, the deciding work is retrieval quality, access control and evaluation against real questions, including the ones the documents cannot answer.
Teams that want an assistant over their own documents, tickets and data usually ask the same first question: should we fine-tune a model on our content, or use retrieval? The short answer is that these techniques solve different problems, and for internal knowledge the answer is usually retrieval first, with fine-tuning added later only if a specific, measurable need appears. Here is the reasoning, without the jargon.
What each approach actually does
Retrieval-augmented generation
RAG splits your documents into passages, indexes them (typically with embeddings and often a keyword index as well), and at question time retrieves the passages most relevant to the question. Those passages are placed in the model's context along with the question, and the model writes an answer grounded in them. The model itself is unchanged; what changes is what it is shown.
Fine-tuning
Fine-tuning continues training a model on examples of inputs and desired outputs, adjusting its weights. It is effective at teaching a model how to behave: a tone of voice, a strict output format, a domain's vocabulary, or a narrow classification task. It is a poor way to store facts that change, because updating a fact means retraining, and a fine-tuned model still cannot tell you which document its answer came from.
A side-by-side comparison
| Consideration | Retrieval-augmented generation | Fine-tuning |
|---|---|---|
| Keeping up with changing content | Re-index the changed documents; answers update immediately | Retrain and redeploy; facts baked into weights go stale |
| Citing sources | Natural: the answer can link to the passages it used | Not available; the model cannot show where an answer came from |
| Respecting user permissions | Filter retrieval by what the current user may see | Everything in the training data is available to every user |
| Handling questions outside the material | Can detect weak retrieval and say "I don't know" | Tends to answer confidently anyway |
| Teaching format, tone or a narrow task | Possible through prompting, with limits | Strong: this is what fine-tuning is for |
| Upfront cost | Engineering time for ingestion, retrieval and evaluation | Curating thousands of quality examples plus training runs |
| Ongoing cost | Longer prompts per query; index maintenance | Hosting or serving a custom model; periodic retraining |
When RAG is the right choice
- The assistant must answer from documents, policies, tickets or records that change over time.
- Users need to see where an answer came from before they trust it.
- Different users are allowed to see different things, so permissions must be enforced at retrieval time.
- The content is too large or too varied to summarise into training examples.
This describes almost every internal knowledge assistant we are asked to build: HR and policy questions, engineering runbooks, customer-support knowledge, contract and procedure lookups.
When fine-tuning earns its place
- A narrow, repetitive task with a stable format: classifying tickets, extracting fields from a known document type, writing in a strict house style.
- Prompting has been tried and measured, and the remaining error rate is about behaviour rather than missing information.
- Latency or cost per request matters enough that a smaller, specialised model is worth the maintenance.
Fine-tuning and RAG are not mutually exclusive. A common mature design uses retrieval for knowledge and a lightly fine-tuned or carefully prompted model for consistent behaviour. Start with retrieval and good prompting; add fine-tuning when you can name the metric it will improve.
The work that actually decides success
Retrieval quality
Most "the AI is wrong" complaints are retrieval failures: the right passage was never shown to the model. Chunking that respects document structure, combining semantic and keyword search, adding metadata filters and re-ranking results all matter more than the choice of model. Measure retrieval separately from answer quality so you can tell which one is failing.
Access control
Permissions must be enforced in the application before retrieval, using the user's identity and the source system's rules. Instructions written into a prompt are not access control; a determined user can talk a model out of them. This is a design requirement, not a feature to add later.
Evaluation with real questions
Collect the questions people actually ask, including ones the documents cannot answer, and review the assistant's behaviour on them before widening the audience. Score groundedness (is the answer supported by the retrieved passages?), correctness, and how the assistant behaves when it should decline. Keep the set and re-run it after every change to the index, prompt or model.
Unanswerable questions
A good assistant says "I could not find this in the approved sources" and points to a human. A bad one improvises. Build the decline path deliberately, test it, and make it easy for users to report wrong answers.
A sensible first project
Choose one defined collection of documents and a small group of users. Collect twenty to fifty real questions. Build retrieval and answering over that collection, with source references and a decline path, and review the results with the users. Only then decide on wider access, automated actions or any fine-tuning. This bounded first release typically takes weeks rather than months, and it produces the evidence you need to justify the next step.
Frequently asked questions
Does fine-tuning make a model "know" our documents?
Not reliably. Fine-tuning shapes how a model responds far more than what it can recall, and it cannot cite where an answer came from. If the goal is accurate answers about specific documents, retrieval is the mechanism that puts the right text in front of the model.
Will our data be used to train someone else's model?
It depends on the provider and the contract terms you select. Business and API tiers of the major model providers generally state that customer data is not used for training, but this should be confirmed in writing and reflected in your architecture, including where indexes and logs are stored.
How much does an internal knowledge assistant cost to run?
Ongoing costs come from model usage per query, the retrieval infrastructure, and keeping the index current as documents change. For most internal tools these are modest compared with the engineering time to build and evaluate the system properly. Ask for both numbers when comparing proposals.