Fine-tuning: when it's the right choice (and when it isn't) [3]
Fine-tuning changes what a model knows permanently. RAG changes what it sees at runtime. They're not interchangeable — and most agentic use cases don't need fine-tuning at all.
![Featured image for "Fine-tuning: when it's the right choice (and when it isn't) [3]"](/images/3.jpg)
Image: PIA26662 — NASA/JPL-Caltech
In the previous post we covered RAG: how to give an agent access to external knowledge without loading it all into context. I also mentioned that fine-tuning is a different approach to the same problem — one that bakes knowledge into the model permanently rather than injecting it at runtime.
This post is about fine-tuning: what it is, the main techniques, and — more importantly — when it actually makes sense to use it. I’ll also tell you that in my work on agentic AI systems, I’ve almost never done it. Here’s why.
What fine-tuning is
Fine-tuning takes a pre-trained model and runs an additional training pass on a specific dataset — an internal document base, a domain-specific knowledge corpus, a set of labeled examples. The model’s weights change. It becomes more specialized.
This is different from RAG in a fundamental way: fine-tuning changes how the model behaves on every call. RAG changes what the model sees on a specific call. One is persistent, the other is transient.
The main techniques
Full fine-tuning
All weights are updated during training. Maximum specialization, maximum cost. Requires significant GPU infrastructure and time. In most applied contexts, the ROI rarely justifies it.
LoRA and QLoRA
LoRA (Low-Rank Adaptation) is the most widely used approach in practice. Instead of updating all the model’s weights, it keeps the original weights frozen and injects two small trainable matrices — A and B — into the existing attention layers. The weight update is approximated as:
W = W₀ + AB
Where W₀ is the frozen original weight matrix and A, B are low-rank matrices with rank r much smaller than the original dimensions. Only A and B are trained — which reduces trainable parameters by orders of magnitude while preserving most of the performance gain.
QLoRA adds quantization on top: the base model is loaded in 4-bit precision, reducing memory requirements dramatically. This is what makes fine-tuning accessible on a single GPU.
RLHF
Reinforcement Learning from Human Feedback is a different category. Instead of a labeled dataset of “correct” outputs, you have human preference data: pairs of responses where annotators indicate which one is better. A reward model is trained on these preferences, and the LLM is then optimized against that reward signal. This is how base models get aligned to produce helpful, non-harmful outputs — GPT, Claude, Gemini all went through variants of this process. DPO (Direct Preference Optimization) is a more recent, simpler alternative that skips the separate reward model.
For most applied practitioners, RLHF/DPO is not something you run yourself. It’s upstream of where most agentic AI work happens.
Why I’ve almost never used fine-tuning in agentic systems
I’ve built multi-agent systems for clients in energy, telco, and public administration. I haven’t fine-tuned a model in any of them. That’s not a gap — it’s a deliberate decision each time, for one or more of the following reasons.
Prompt engineering often gets you there
Before investing in fine-tuning, ask: can a vanilla model with a well-crafted system prompt do this task adequately? More often than not, the answer is yes. Prompt engineering is cheap, fast to iterate, and model-agnostic. Fine-tuning is expensive, slow, and tied to a specific model version.
Knowledge bases change
If the domain knowledge you want to teach the model is subject to frequent updates — regulatory changes, product catalog changes, policy revisions — fine-tuning is the wrong tool. Every update means retraining. RAG handles dynamic knowledge by design: update the vector database, the model picks it up immediately.
Source attribution matters
This is the argument that comes up least often but that I find most compelling. When an LLM responds to a user using information from a fine-tuned model, you cannot trace which training document produced that output. The information is diffused across millions of weight updates, potentially blended from multiple sources.
If your business context requires showing users the source of an answer — the contract clause, the regulation article, the technical manual page — fine-tuning can’t give you that. RAG can, because the retrieved chunks are explicit.
Hallucinations become harder to detect
This is counterintuitive. A model fine-tuned on highly technical or vertical domain data adopts the vocabulary and style of that domain. When it hallucinates, it hallucinates fluently and convincingly within that register. A vanilla model hallucinating outside its comfort zone is easier to spot — the output tends to be generic or visibly uncertain.
This doesn’t mean fine-tuning causes more hallucinations. It means the ones it produces are harder to catch, which raises the stakes for evaluation.
Model obsolescence
If you invest significant effort in fine-tuning a specific model version, and six months later a better base model is released, you face a choice: stay on the older model or redo the entire fine-tuning process. In a field that moves as fast as LLMs, this is a real cost to factor in. RAG pipelines are model-agnostic by design — swapping the underlying model is usually a configuration change, not a retraining project.
When fine-tuning is the right choice
None of the above means fine-tuning is never the answer. There are cases where it’s clearly the better tool:
- Style and tone adaptation — teaching a model to communicate in a specific voice, register, or format that’s consistent across all outputs
- Task-specific behavior — when the model needs to reliably follow a particular output structure (JSON schema, classification labels, specific response format)
- Latency-sensitive high-volume tasks — at sufficient scale, fine-tuning can be significantly more cost-efficient than RAG per query
- Static, controlled knowledge — if the domain knowledge is stable and fully under business control, fine-tuning avoids the runtime overhead of retrieval
The decision framework
Before choosing between fine-tuning and RAG, ask:
| Question | If yes → |
|---|---|
| Does the knowledge change frequently? | RAG |
| Is source attribution required? | RAG |
| Is the knowledge base outside business control? | RAG |
| Could a new model make this obsolete? | RAG |
| Is the task about style, format, or behavior? | Fine-tuning |
| Is the knowledge stable and bounded? | Fine-tuning |
| Are you running millions of similar queries? | Fine-tuning |
In most agentic AI contexts, the first four conditions apply. Which is why RAG — or a combination of good prompting and RAG — tends to win. Fine-tuning is not wrong; it’s just often unnecessary.
What’s next
We’ve covered the foundational building blocks — LLMs, agents, context, RAG, fine-tuning. The next posts will go deeper into multi-agent architecture: how pipelines are actually structured, how agents hand off tasks to each other, and where things tend to break.
If you have questions or want to suggest a topic, reach out.