RAG vs fine-tuning: which one does your AI problem actually need?

Two ways to make a language model useful on your own material. Retrieval-augmented generation — RAG — looks your documents up at question time and hands the relevant pieces to the model along with the question. Fine-tuning adjusts the model itself on examples until it behaves differently. They solve different problems, and confusing them is the most expensive mistake in this area.

The test that decides it

Are you giving it knowledge, or giving it manners? Facts, documents, policies, meeting transcripts, product catalogues — anything that changes over time and needs to be quoted accurately — is retrieval. Tone, format, a house style, a consistent way of structuring an answer — behaviour that does not change weekly — is fine-tuning.

The meeting system on this site is retrieval, and deliberately so: what somebody said in a meeting is a fact, it must be quoted exactly, and it must be attributable back to who said it and when. No amount of fine-tuning makes a model reliably recall a specific sentence from a specific call — but retrieval hands it that sentence and the model simply reads it back.

Why RAG is usually the right first move

• Your content changes. Updating an index takes seconds; retraining does not.
• Answers can cite their source, which is what makes the output checkable rather than trusted blindly.
• It fails visibly. If the retrieval finds nothing, you can say so — a fine-tuned model will confidently invent instead.
• It is far cheaper to build, and far cheaper to be wrong about.

What building retrieval actually involves

The parts are unglamorous and this is where the quality is won or lost. Documents are split into chunks small enough to be precise and large enough to keep their meaning. Each chunk is embedded and indexed. At question time the question is embedded too, the closest chunks come back, and they go to the model with an instruction to answer only from them and to say so when they do not contain the answer.

Almost every disappointing RAG system fails in the chunking, not the model: split a document badly and you retrieve half a sentence with the qualifier missing. That is the work, and it is why "we will just plug in a vector database" underestimates the job.

The failure modes to design for

• Retrieval returns nothing relevant and the model answers anyway — the fix is an explicit instruction to decline, and a system that flags it rather than hides it.
• The right document exists but the wrong chunk is retrieved, so the answer is confidently incomplete.
• Sources are not shown, so nobody can tell a correct answer from a plausible one.
• A fine-tune is used to paper over a retrieval problem, which makes wrong answers more fluent rather than more correct.

Cost and speed, honestly

Retrieval adds a step before the model runs, so answers arrive slightly slower than a bare prompt, and every answer carries the retrieved text through the model, which costs tokens. In exchange you skip training entirely and you can change what the system knows by re-indexing a folder. For nearly every business case that trade is overwhelmingly the right one — the latency is measured in a fraction of a second and the alternative is measured in a retraining cycle.

When fine-tuning earns its cost

When the task is narrow, high-volume and stylistic — classifying thousands of items into your own categories, or producing output in a rigid format every time — fine-tuning can beat a long prompt on both cost and consistency. That is a real case. It is just far rarer than the number of teams reaching for it.

And the two are not exclusive. The common mature answer is retrieval for the facts, with prompt discipline — or a light fine-tune — for the behaviour on top.

Facts are retrieval. Behaviour is training. Most problems are facts.