Fine-Tuning or RAG: Train an AI Model on Your Company's Data or Let It Search Your Documents

If you want an AI model to answer questions from your company's documents, you usually do not need to train it on them. Retrieval-augmented generation, or RAG, searches your documents each time a question comes in and passes the matching passages to the model along with the question. Fine-tuning means training an existing model further on examples you prepare. Research papers and vendor guides say it is better at teaching a format or a tone than at teaching new facts. A test with real questions from your business shows which one you need, or whether you need both.

What is the difference between fine-tuning and RAG?

A large language model (LLM), the kind of AI behind ChatGPT, learns what it knows from the text it was trained on. Documents that never left your company were not part of that text, so the model has to get them some other way.

With RAG, your documents stay outside the model in a search index, a catalogue of passages built from your files. When someone asks a question, the system finds the passages that match and adds them to the prompt, the text the model receives. Because the system knows which passages it used, it can show the source document next to each answer.

Fine-tuning trains the model further on examples, each pairing a question or task with the answer you want. The result is a new version of the model. What it learned is spread across the model's internal numbers, and no file shows where a given fact came from.

OpenAI's guide “Optimizing LLM Accuracy”, read on 23 October 2026, explains the difference with an exam. RAG is like having “the textbook with you” to look things up. Fine-tuning is like attending class for months and learning from “many repeated examples”.

The guide adds that the two methods “are additive, not exclusive”. It also says “many of our largest customer deployments at OpenAI were done using only prompt engineering and RAG”. Prompt engineering means writing better instructions and examples for the model.

Can we fine-tune a model on our documents so it knows our business?

Published research says that training a model on your documents does not reliably teach it the facts of your business. In a study first posted in December 2023, Oded Ovadia and three co-authors compared RAG with fine-tuning a model on the plain text of documents. Their abstract says that “RAG consistently outperforms it” and that models “struggle to learn new factual information” this way. It adds that showing the model “numerous variations of the same fact” during training could help.

A 2024 paper by Zorik Gekhman and six co-authors tested fine-tuning on questions paired with answers. Examples that contain new facts “are learned significantly slower” than examples that match what the model already knows. Once the model did learn them, it became more likely to give factually wrong answers. The authors write that their results support the view that models “mostly acquire factual knowledge through pre-training”, meaning the original training done by the model's maker, and that fine-tuning “teaches them to use it more efficiently”.

For a business, facts that change, such as prices or policy wording, belong in documents the system searches. A fine-tuned model keeps what it learned on the day it was trained, and changing that takes another training run.

When is fine-tuning worth it?

OpenAI's guide says to work on how the model behaves, rather than on the information it is given, when it is “producing inconsistent results with incorrect formatting” or when its tone or style of speech “is not correct”. Better instructions and examples in the prompt come first, and fine-tuning is the next step. The guide also names efficiency as a reason to fine-tune, meaning the same accuracy with a smaller model or with fewer tokens. A token is a piece of a word, and model providers charge per token.

For example, a model that reads invoices must return their data in the exact layout your accounting system accepts. Replies drafted for customers may need to sound like your support team. A smaller fine-tuned model can also cost less to run than a large general one, if it reaches the accuracy you need.

The training examples have to look exactly like the real work. The guide advises starting with “50+ examples”, then testing, and adding more only if the errors that remain are about how the model behaves. Someone who does this work every day has to write or check those examples, so include their hours in the budget along with the engineering time.

Our document set is small. Do we need RAG at all?

With a small set of documents, you may not need RAG at all. Anthropic, the company behind the Claude models, wrote in a post dated 19 September 2024: “If your knowledge base is smaller than 200,000 tokens (about 500 pages of material), you can just include the entire knowledge base in the prompt that you give the model, with no need for RAG or similar methods.”

Each model has a limit on how many tokens it can read at once. Prompt caching lets an application reuse text it sent in earlier requests, such as the same set of documents. The post, read the same day as the guide, says prompt caching cuts costs “by up to 90%” with Claude. OpenAI's guide warns that models “can struggle to maintain attention across very large prompts”, so test at the size you plan to use.

Sending every document with every question still costs more than sending a few passages, and the cost grows with each page you add. A pilot can start with a small set, such as the manuals for one product line, and move to RAG when the documents no longer fit in the prompt.

What can go wrong with RAG?

A RAG system can fail before the model writes a word, when the search returns the wrong passages or too many of them. The OpenAI guide describes both cases. The system can supply “the wrong context, so the model can't possibly answer”. Too much irrelevant text is the other risk, because it “drowns out the real information and causes hallucinations”, meaning answers that sound right but are false.

Many RAG systems search by meaning rather than by exact words, and that can miss exact codes. Anthropic's post gives the example of a support question about error code TS-999. Such a search might find pages about error codes in general and miss the one for TS-999. The post suggests combining it with keyword search, which looks for the exact text.

So test the search on its own. For each test question, check whether the right passage came back before you judge the answer. If it did not, OpenAI's guide points to the search as the thing to fix.

Which keeps our data under control, RAG or fine-tuning?

Both can run on your own servers. With RAG, your documents sit in an index you control, and the model sees only the passages picked for each question. Fine-tuning builds the training examples into the model files, and what the model learned from them can reappear in its answers.

RAG makes access rights easier to enforce, because the search can leave out documents that the person asking is not allowed to see. Microsoft describes one way to do this in its Azure AI Search service. Its page on the security filter pattern, read the same day, describes a filter that “trims search results based on a string containing a group or user identity”. The page offers this filter for systems that cannot use the service's built-in access control lists.

The same page says there is “no authentication or authorization” in that filter itself, so your application has to pass the right user or group with every search. A fine-tuned model has no equivalent. What it learned from one department's examples can appear in answers to any user of that model.

The European Data Protection Board (EDPB), made up of representatives of national data protection authorities and the European Data Protection Supervisor, adopted Opinion 28/2024 on 17 December 2024. It says that “AI models trained on personal data cannot, in all cases, be considered anonymous”, because personal data “may still remain ‘absorbed’ in the parameters of the model”. Parameters are the internal numbers in which a model stores what it learned. Whether a given model is anonymous, the opinion adds, has to be assessed case by case.

If you fine-tune on support tickets or customer files, assume that the model itself may hold personal data. Ask how you would handle a request to delete one person's records. With RAG, once a document is removed and the index is updated, it is no longer found for new questions. Copies kept in logs, caches or saved chat histories need their own clean-up. Rules differ by country, and this article is not legal advice, so check your case with a lawyer.

How long will a fine-tuned model keep working?

A fine-tuned model is built on a base model, the general model you started from. If you fine-tune through a cloud service, the provider decides how long that base model, and your version of it, stays available. OpenAI's API pricing page, read the same day, says “OpenAI is winding down the fine-tuning platform.” It adds that fine-tuned models stay available until their base models are deprecated, OpenAI's term for models it is retiring.

OpenAI's deprecations page gives the dates. From 7 May 2026, organisations that had never run fine-tuning could no longer start. From 2 July 2026, the same applied to organisations that had not used a fine-tuned model in the previous 60 days. On 6 January 2027, “Active existing customers will no longer be able to create new fine-tuning jobs.”

The same page also sets end dates for the models themselves. It lists 23 October 2026 as the shutdown date for fine-tuned versions of gpt-4.1-nano and five other models. GPT-4o, GPT-4o mini, GPT-4.1 and GPT-4.1 mini follow on 14 January 2027, and “Fine-tuned versions of these models follow the same shutdown schedule as their underlying base models.”

Open-weight models, whose makers publish the model files for anyone to run under a licence, can still be fine-tuned on your own hardware. OpenAI's gpt-oss-120b is one, and its page on the Hugging Face model site, read the same day, lists the Apache 2.0 licence. The page says the model “can be fine-tuned on a single H100 node”, meaning one server with Nvidia H100 GPUs, a type of chip the page names for running the model.

Whether the model runs in a provider's cloud or on your own hardware, a new base model means another round of fine-tuning and testing. A RAG system needs re-testing too when the model that writes the answers changes, but the index and your documents stay as they are.

What do RAG and fine-tuning cost to build and run?

No figures are given here, because they depend on your volumes and setup. The parts of the bill are different, though.

For RAG, the build work is preparing documents so they can be searched well, building the index and testing the search. Running costs include the search service and longer prompts, since each question carries the passages found for it. Re-indexing can run automatically when a file is added or changed.

With fine-tuning, much of the cost can be staff time, because people who know the task write and check the training examples. Then come the training runs and the servers or service that host the new model. A new base model or a new task means paying for that work again.

Some systems need both. If early tests show problems with facts as well as format, OpenAI's guide says “it's likely you may end up with fine-tuning + RAG”. You then pay for each.

Should we choose RAG or fine-tuning, and what should we ask a firm that offers to build it?

Start with a test set made of real questions or documents from your work, each with an answer a person has checked. OpenAI's guide says that a set of “20+ questions and answers”, once you have looked at why the failures happen, gives you “the right baseline” for more advanced methods. The article on a stalled AI pilot explains how to build one.

Then try a well-written prompt with RAG and group the failures by cause. A missing passage means the search needs work. Sometimes the right passage came back and the model still misread it or answered in the wrong format. For that case, the guide suggests better instructions first, and fine-tuning if a few worked examples in the prompt improve the answers.

Put the same questions to every firm that proposes a design:

• Which results on our test set led you to propose fine-tuning?
• How will the system keep staff from seeing documents they are not allowed to open?
• If you fine-tune, who writes the training examples, and what happens when the base model is retired?
• After launch, how do we remove a document or one person's data?

Where does amBrain fit?

The one language model project amBrain describes in public is this: “We have taken an LLM integration to production inside a client's FinTech perimeter: extracting and normalising unstructured broker and venue notices — corporate actions, instrument and margin changes — into structured records the trading system consumes.”

This article is not a case study and describes no client work. The quoted line does not say whether that system uses RAG, fine-tuning or neither, which model it uses or where that model runs. The examples in this article are general and are not taken from that project or any other amBrain work.

The companies, public bodies, models and research papers named here are cited from their published pages, and the article claims no relationship between amBrain and any of them. It gives no amBrain prices, timelines or terms.

amBrain has been building software since 2019. It works in three formats: full delivery, a dedicated team, or engineers embedded in your team. Its line on ownership is “The client keeps full ownership of the product and the code, except our reusable components.” If amBrain is on your shortlist, give it the same questions as any other firm.