Should we fine-tune a model on our company data?

Usually not as a first step. If the goal is for the AI to know your facts, such as prices, policies or product details, retrieval-augmented generation (RAG) is the better tool, because the facts stay in documents you can update, cite and permission. Fine-tuning is better suited to teaching a consistent style, format or narrow task once prompting and retrieval have been measured and found short.

What is the difference between RAG and fine-tuning?

RAG leaves the model unchanged and gives it relevant passages from your content at the moment it answers. Fine-tuning changes the model itself by training it further on example inputs and outputs. RAG is suited to facts that change and need a source; fine-tuning is suited to behaviour such as tone, output format or classification.

How many examples do you need to fine-tune a model?

It depends on the task and the provider, but you generally need a substantial set of high-quality, reviewed examples of the exact input and output you want, often hundreds or more. Fine-tuning on a small or messy set tends to teach the model the mess, which is why data preparation is often the largest part of the cost.

Fine-Tuning vs RAG vs Prompting: How to Make an AI Model Know Your Business

Q: Can fine-tuning and RAG be used together?

Yes. A fine-tuned model is usually still given a prompt and, where facts matter, still given retrieved context. Fine-tuning sits on top of good prompting and retrieval rather than replacing them.

TL;DR: If you want an AI model to "know your business", you have three tools, and they solve different problems. Prompting tells the model how to behave on this request. Retrieval-augmented generation (RAG) hands it the facts it needs at the moment it answers. Fine-tuning changes the model itself by training it on your examples, which teaches it a style, a format or a narrow skill, but is a poor way to teach it facts. Most business AI features should start with good prompting plus RAG, prove themselves against an evaluation set, and only consider fine-tuning when there is a specific, measured gap that the first two cannot close. If a vendor proposes fine-tuning first, ask what problem it solves that retrieval does not.

"Can we train it on our data?" is one of the first questions we hear when a business starts planning an AI feature. It is a reasonable question, and the honest answer is usually "you probably do not need to, and if you did it would not do what you expect." The word "train" covers several very different techniques with different costs, risks and results.

This guide is for business owners and product leaders deciding how to make a model useful on their own material. It explains what each approach actually changes, when each one is the right call, and what to ask before you pay for any of them.

The three tools, in plain terms

Prompting: instructions for this request

A prompt is the text the model receives before it answers: the role it plays, the rules it follows, the tone it uses, the format it returns, and examples of good output. Changing the prompt changes the behaviour immediately, costs nothing to deploy, and is easy to undo.

Prompting is more powerful than it sounds. A large share of "the AI is not doing what we want" problems are solved by clearer instructions, a few worked examples, and a defined output structure. It is always the first lever to pull.

RAG: facts at the moment of answering

Retrieval-augmented generation means searching your own content for the passages that matter to a question and giving them to the model alongside it. The model writes the answer; your documents supply the facts. We cover the build in detail in building an internal AI knowledge base.

The key property is that the knowledge lives outside the model. When a price, policy or opening hour changes, you update the document and the next answer reflects it. You can also show the source an answer came from, and you can enforce permissions at retrieval time so a user never sees passages they are not allowed to read.

Fine-tuning: changing the model's habits

Fine-tuning takes an existing model and trains it further on a set of example inputs and the outputs you want. The result is a modified model that leans toward those patterns without being told every time.

What fine-tuning is good at is behaviour: a consistent house style, a strict output format, a classification task with your own categories, or making a smaller and cheaper model perform a narrow task as well as a larger one. What it is bad at is reliably memorising facts. A fine-tuned model may reproduce something it saw in training, or it may produce something that sounds just like it and is wrong, and it cannot tell you which.

Why fine-tuning is the wrong tool for facts

This is the misunderstanding behind most "train it on our data" requests, so it is worth being direct about.

Facts change, and a fine-tuned model is frozen. Every update to your pricing, catalogue or policies means another training run, another round of testing and another deployment. With RAG it means editing a document.
There is no source to show. A RAG answer can point to the paragraph it used. A fine-tuned answer comes from the model's weights, so nobody can check where it came from.
Permissions do not exist inside a model. Once restricted information is trained in, any user of that model can potentially draw it out. Telling the model to keep secrets is not a security control. This matters even more when the system can take actions, as we cover in AI agents with write access.
Errors are harder to fix. If a RAG system gives a wrong answer, you can usually find the bad or missing document. If a fine-tuned model does, the fix is more data and another training run, with no guarantee the mistake is gone.

For a business, the practical rule is simple: facts go in retrieval, behaviour goes in the prompt, and fine-tuning is reserved for behaviour that the prompt cannot hold.

When each approach is the right call

Start with prompting when

The task is clear but the output is inconsistent in tone, length or format.
The model has the general knowledge it needs and just needs direction.
You are early and still learning what good looks like.

Add RAG when

Answers depend on your own documents, records, policies or catalogue.
That information changes, even occasionally.
Users need to see or trust the source.
Different users are allowed to see different information.

Most of the AI features we build sit here. The voice agents we built for CallGuard AI and CallSetter AI have to know each business's services, hours and booking rules, and that information changes. The multilingual voice and SMS intake behind Fortell AI, which serves Community Action Agencies in more than 100 languages, has to answer against real program rules. In all of these, the business facts belong in content the system can retrieve and the team can update, not baked into a model.

Consider fine-tuning when

You have a narrow, repetitive task, such as classifying messages into your own categories or extracting fields in a fixed format, and prompting alone is not consistent enough at the volume you run.
You want a smaller, faster or cheaper model to match a larger one on that single task, which can matter for both cost and response time. See AI cost control and, for voice, voice AI latency.
You have a style that is hard to describe but easy to show, and enough good examples to show it.
You have hundreds or more high-quality examples of the exact input and output you want, reviewed by someone who knows the business.

That last condition rules out more projects than any other. Fine-tuning on a small or messy set of examples tends to teach the model the mess.

The order we recommend

Define what good looks like and build an evaluation set from real inputs. Without it, you cannot tell whether any of these techniques helped. We explain how in evaluating AI features.
Get the prompt right, including examples and an output structure, and measure against the set.
Add retrieval for anything that depends on your own information, and measure again. Most quality problems at this stage are content problems: duplicate, outdated or missing documents.
Choose the model deliberately. Sometimes a different model closes the gap with no training at all. See choosing an AI model.
Only then consider fine-tuning, for a specific measured gap, with a clear target score and a comparison against the untuned setup.

The steps are cumulative. A fine-tuned model is usually still given a prompt and, where facts matter, still given retrieved context. Fine-tuning does not replace the other two; it sits on top of them.

Hidden costs of fine-tuning

The training run itself is often the smallest cost. The ones that surprise teams later are:

Preparing training data. Collecting, cleaning and reviewing examples is real work, and most of it needs someone who knows the business, not just an engineer.
Retraining when the base model changes. Providers update and retire models. A fine-tune built on a retired model has to be rebuilt on its successor and retested.
Vendor and hosting lock-in. Some fine-tunes can only run on the provider that trained them. Ask where the tuned model lives, who can access it, and whether you can take it with you. Our view on ownership is in software ownership and handover.
Data handling. Your examples are sent to whoever performs the training. Check what the provider does with them and whether personal or regulated data is involved.
Ongoing evaluation. A tuned model still needs the evaluation set run on every change, and the set needs maintaining.

None of these are reasons to avoid fine-tuning when it is the right tool. They are reasons not to choose it by default.

What this looks like outside customer-facing chat

The same thinking applies to product features. For an app like LectureNotes AI, which records lectures and produces summaries and outlines, the important qualities are faithfulness to what was said and a consistent outline structure. Faithfulness comes from grounding the output in the transcript; structure is first a prompting job. For a consumer product like Lifemaxxing AI, which builds personalised plans, personalisation comes from the user's own data passed in at request time, not from training a model on every user.

In both cases, the useful question is not "should we train a model?" but "where should this knowledge or behaviour live so it stays correct and can change?" If you are adding AI to a product that already exists, the wider plan is in adding AI to an existing product.

Questions to ask before you approve fine-tuning

What specific, measured problem does fine-tuning solve that prompting and retrieval have not?
What does the evaluation set show for the untuned setup, and what score are we aiming for?
How many training examples do we have, where did they come from, and who reviewed them?
Is any of our factual or restricted information going into the training data?
Where will the tuned model run, who can access it, and can we move it?
What happens when the base model is updated or retired?
How will we keep facts current if they live in the model?

The bottom line

Prompting shapes how a model behaves, retrieval gives it your facts, and fine-tuning changes its habits. For most business AI features, the right build is a strong prompt, clean retrieval over well-maintained content, and an evaluation set that proves it works. Fine-tuning is a real and useful tool, but it is a targeted one, best used after the simpler options have been measured and found short. When someone offers to train a model on your data, the most valuable thing you can do is ask what it will fix.

Planning an AI feature and not sure whether it needs retrieval, fine-tuning or just better prompts? Book a demo and we will scope the right approach with you. See our work: CallGuard AI, CallSetter AI, Fortell AI, LectureNotes AI, Lifemaxxing AI and more.