RAG vs. fine-tuning: a decision framework for LLM apps

RAG gives a model facts at request time. Fine-tuning teaches it patterns at training time. RAG vs. fine-tuning gets framed as a rivalry, but the two fix different problems, and most teams pick one before they know which problem they have. A support bot that invents a refund policy has a knowledge gap. A bot that knows the policy but answers in the wrong structure has a behavior gap. Retrieval does nothing for the second, and training is a poor fix for the first.

This guide covers how each technique works and fails, why prompting comes before both, a decision framework with a worked cost example, and the patterns for combining them. For background on weights, context windows and training stages, start with how large language models actually work.

RAG vs. fine-tuning at a glance

The table compares the two on the seven axes that decide most projects. The RAG and fine-tuning columns are two lists of fixes for different problems.

Two rows drive most decisions. If the knowledge changes faster than you can retrain, only the RAG column keeps up. If the failure is in how the model writes or works through your task, only the fine-tuning column touches it.

How RAG works and where it breaks

A model's weights stop at its training cutoff, and they never included your private data, so a question about last week's pricing change gets a fluent guess. The key idea behind RAG is simple: look the answer up first, then let the model write from what you found.

The pipeline

Offline, you split documents into chunks, turn each chunk into a vector with an embedding model, and store the vectors in an index. At query time you embed the question, retrieve the top-k nearest chunks, assemble the prompt and generate. Many systems add a second model that re-scores each chunk against the query before generation. Many also run hybrid retrieval, which merges keyword search (BM25) with dense vectors, because exact identifiers such as part numbers match better as strings.

The costs RAG moves onto every request

RAG is cheap to set up, but you pay per call. Each retrieved chunk lengthens the prompt, which raises prefill compute and time to first token (TTFT) on every request; the mechanics are in prefill and decode latency. A longer prompt also occupies more KV cache memory per request, which means fewer concurrent requests fit on each GPU. The retrieval hop adds an embedding call, an index lookup and perhaps a rerank, all before prefill starts. With 5 chunks of 600 tokens each, that's 3,000 extra input tokens on every call.

Common RAG failure modes

Most bad RAG answers trace back to one of these:

• Retrieval misses: The right chunk never reaches the prompt, because chunking splits the answer in two or because an off-the-shelf embedding model doesn't place your domain jargon near the user's phrasing. The generator can't recover from this.
• Lost in the middle: Extra chunks added for safety can backfire. Liu et al. (2023) found, on multi-document question answering and key-value retrieval, that accuracy was often highest when the relevant information sat at the start or end of the context. It degraded significantly when that information sat in the middle, even for long-context models.
• A generator that ignores good context: Retrieval succeeds, but the model answers from what it learned in pretraining, blends in an irrelevant chunk, or drops your citation format. This is a behavior failure inside a knowledge system, and it's where training helps RAG.

RAG is not a hallucination cure. It narrows what the model should say, but it can't force the model to say it.

What fine-tuning changes and what it can't do

What changes in the model

Fine-tuning runs more gradient steps on a pretrained model using your data. Supervised fine-tuning (SFT) trains on input/output pairs so the model learns to imitate the outputs. Preference tuning like Direct Preference Optimization (DPO) trains on chosen/rejected pairs, which suits qualities you can rank more easily than you can write. Either method can update every weight (full fine-tuning) or train a small adapter on top of a frozen base (LoRA). LoRA trains 0.1% to 1% of the parameters a full run would (see LoRA vs. full fine-tuning), which is why it's the usual starting point. Data prep, hyperparameters and method choice are covered in how to fine-tune an LLM.

What it's good at

Fine-tuning earns its cost in five situations:

• Consistent format and schema: A model picks up an output shape from many examples more reliably than from instructions.
• House style and tone: The same holds for voice, where a three-page style prompt still drifts after a few turns.
• Domain task skill: Classification, extraction and routing against your own taxonomy.
• Suppressing base behavior: Over-refusal, verbosity and reflexive disclaimers.
• Smaller, cheaper inference: A small model with a one-line prompt replaces a large model with a long one, which cuts cost per call and TTFT.

Why it's a poor way to add facts

Ovadia et al. (2023) compared RAG with unsupervised fine-tuning (continued training on raw text) across knowledge-intensive tasks. RAG consistently outperformed it, both for knowledge the model saw in pretraining and for entirely new knowledge. Gekhman et al. (2024) built a controlled closed-book QA setup and varied the share of supervised fine-tuning examples that introduced facts new to the model. The model learned those examples significantly slower than examples consistent with what it already knew, and as it eventually learned them, its tendency to hallucinate rose linearly. Both setups are narrow. Treat them as direction.

The operational reasons agree. A fine-tuned model can't cite a source, it's stale the day your data changes, and each retraining round risks catastrophic forgetting in continued fine-tuning, where earlier skills degrade as new data is learned.

So retrieval is usually the stronger tool for loading facts. Fine-tuning stays the stronger tool for behavior, latency, and cost. One task-scoped example from August 2025: Parsed reported that Gemma 3 27B, fine-tuned on tens of thousands of examples, outperformed Claude Sonnet 4 by 60% on pass rate against Parsed's own clinical-note evaluation harness (small open models fine-tuned for healthcare scribing). That result is about note quality and format on one task, not about teaching the model medicine.

Prompting comes first

Prompting is the baseline and the control group for every other option. It's effectively free because you already pay for inference, a change takes seconds to test, and a bad prompt is fully reversible. Few-shot examples and schema-constrained decoding close many "we need fine-tuning" gaps before any training run. If the complaint is malformed JSON, try structured outputs and JSON mode first.

The escalation ladder follows the gap. Improve the prompt. Add retrieval if what's missing is knowledge. Fine-tune if what's missing is behavior the prompt can't lock in, or if a long prompt costs too much at your volume. Together's docs go deeper on that last step in when to fine-tune vs. prompt, and its rule of thumb is worth keeping. If a good prompt gets you 90% of the way in a week, fine-tuning buys the last 10% at roughly 10× the cost in time and maintenance. Sometimes that trade is worth it. Spend that week on the prompt first.

A decision framework for RAG, fine-tuning, or both

Diagnose the gap first

Build a labeled eval set of 50 to 200 examples, then run the base model with your strongest prompt against it. Read the failures and sort them. Wrong or missing facts are a knowledge gap, which points to RAG. Right facts in the wrong shape, tone, or sequence of steps are a behavior gap, which points to fine-tuning. A real mix of both points to a hybrid. No eval set means no decision, because you can't tell which gap dominates or whether a fix worked.

Later, compare prompt-only, RAG, fine-tuned, and hybrid variants on the same held-out set. Track retrieval recall, faithfulness to the retrieved text, task accuracy, latency, and cost per request as separate numbers, because a single blended score hides which component regressed. The methods are in the guide to LLM evaluation.

Six checks that settle most cases

Work through these six checks in order:

• Changing or private data: If answers depend on either, you need RAG, whatever else you add.
• Citations and audit: If answers must point to a source, you need RAG, because weights can't cite.
• Format, style, or procedure failures: These are behavior gaps, so retrieval won't move them.
• Training data: Fine-tuning needs hundreds of high-quality labeled examples at minimum, usually thousands, or a reliable way to generate them.
• Volume and latency budget: High volume and a tight TTFT target favor a short prompt on a small fine-tuned model. Low volume favors prompting.
• Ownership: Someone maintains the index, someone maintains the training pipeline, and a fine-tune must be retrained when its base model is upgraded.

Decision matrix

Cost and latency: a worked example

The prices below are round example numbers chosen for easy arithmetic, not a quote from any price list. Suppose a 70B-class model on a per-token API costs \$1.00 per 1M input tokens and \$3.00 per 1M output tokens. Each request carries 3,000 tokens of instructions and few-shot examples, 3,000 tokens of retrieved context, and 300 output tokens.

Cost per request ≈ (6,000 × \$1.00 + 300 × \$3.00) / 1M = \$0.0069

Now fine-tune an 8B-class model that hits your quality bar, so the 3,000 instruction tokens shrink to one line. The retrieved context stays, because facts belong in the index. On Together, fine-tuned models are served on dedicated endpoints, which bill per minute of GPU time instead of per token. Assume one H100 at \$3.00 per hour, or \$72 per day running around the clock, and a \$500 training run.

At 50 calls a day the fine-tune never pays back. At 50,000 it saves ~$273 a day and recovers the training run in about 2 days, assuming one GPU holds your peak traffic. The shorter prompt also means less prefill work, so TTFT drops. Building the dataset and eval set usually costs more than the GPU time, and it is missing from this table.

Combining RAG and fine-tuning

Most hybrids follow one of these patterns:

• Fine-tune the generator on retrieved context: Train on the question, the retrieved documents (including irrelevant distractors) and the answer, so the model learns to use the relevant passage and ignore the rest. This is the RAFT recipe (Zhang et al., 2024), which improved in-domain RAG results on PubMed, HotpotQA, and Gorilla datasets. A June 2024 experiment, RAG fine-tuning for a code assistant, used a similar recipe. Mistral 7B Instruct v0.2, trained on questions paired with retrieved code snippets, improved on the untuned model on 4 of 5 codebases, measured by HitRate with retrieval on.
• Fine-tune the retriever or add a reranker: Domain jargon and internal identifiers break off-the-shelf embeddings. Contrastive fine-tuning of the embedding model, or a reranker over a wider top-k, often lifts recall for less effort than touching the generator.
• Fine-tuned behavior on top of retrieval: Train for output format, citation style and tone, and keep the facts in the index. The split is clean, because each half does only what it's good at.

A hybrid means two pipelines and two eval surfaces, and a regression can come from either one. Start with one technique, and add the second only against a gap you've measured.

Conclusion

Retrieval decides what the model reads. Training changes how it writes and works through the task. Diagnose the gap with an eval set, fix knowledge with retrieval, fix behavior with training, and run both only when you've measured both gaps. Keep the prompt-only baseline around as the control whichever way you go. If the diagnosis points to training, you can fine-tune models on Together AI and serve the result on a dedicated endpoint.

FAQ

Is fine-tuning better than RAG? Neither is better in general, because they fix different gaps. Use RAG for knowledge that changes or must be cited. Use fine-tuning for behavior, output format and cost at high volume.

Is fine-tuning still relevant? Yes, for behavior, latency and cost, because a small fine-tuned model with a short prompt can replace a large model with a long one. For teaching a model new facts it is the weaker choice, and retrieval beat unsupervised fine-tuning in the comparison cited above.

What's the difference between prompting, RAG, and fine-tuning? Prompting changes the instructions, RAG changes the context, and fine-tuning changes the weights. Try them in that order, because each step costs more to build and iterate on than the one before.

Does fine-tuning reduce hallucinations? It can reduce format and task errors. Training on facts the model doesn't already know can increase hallucination, as Gekhman et al. (2024) found in closed-book QA. Grounding answers with retrieval and measuring faithfulness is the more direct lever.