TL;DR: You cannot make a language model incapable of making things up, but you can build a product where invented answers rarely reach a customer and are caught when they do. The reliable pattern has five parts: give the model the facts instead of asking it to remember them, let code handle anything that must be exact (prices, dates, totals, availability), constrain what the output is allowed to look like, give the model a clean way to say "I don't know", and check claims against the source before they are shown or acted on. A better model reduces the problem. Product design is what controls it.
"Will it hallucinate?" comes up on nearly every AI project call, usually after the buyer has seen a demo answer something confidently and wrongly. The honest answer is yes, sometimes, on any model. The useful question is a different one: when it does, what happens next? A wrong sentence in a brainstorming tool costs nothing. A wrong price quoted to a caller, a wrong total in a report or an invented policy in a support reply costs real money and trust. This article is about building for the second kind.
Why models make things up
A language model produces the most plausible continuation of what it has been given. Most of the time plausible and true line up. When the model lacks the fact it needs, it does not stop. It produces something that sounds like the fact.
Three situations cause most invented answers in business products:
- The fact was never provided. The model is asked about your opening hours, your refund policy or a specific customer's order, and none of that was in its context. It fills the gap from general patterns.
- The fact was provided but buried or contradicted. Two documents disagree, the relevant passage was cut off, or the retrieval step pulled the wrong page. The model blends what it has.
- The task requires exactness the model is bad at. Arithmetic across many rows, counting, date calculations and long exact strings like account numbers are where fluent text and correct output part ways.
Each of these has a different fix, which is why "use a better model" or "write a stricter prompt" alone rarely solves it.
Pattern 1: Give it the facts, do not ask it to remember
The most effective single change is to stop relying on what the model learned in training and instead supply the relevant facts with every request. This is retrieval, often called RAG, and the trade-offs against fine-tuning are covered in fine-tuning vs RAG vs prompting. The short version: fine-tuning changes how a model behaves, not what it reliably knows, so it is the wrong tool for facts.
Retrieval only helps if the right facts arrive. In practice most "the AI made it up" reports trace back to retrieval returning the wrong passage, an outdated document, or nothing at all. That is a content and pipeline problem more than a model problem, and building an internal AI knowledge base covers it in detail: conflicting documents, knowledge nobody wrote down, and keeping the index current.
For structured business data, retrieval often means a direct lookup rather than a document search. If the question is about an order, fetch the order record and pass it in. A model reading an actual record is far more reliable than one searching a pile of text for something that looks related.
Pattern 2: Let code handle anything that must be exact
Some outputs have exactly one right answer: a price, an appointment slot, an invoice total, a policy number, a tax figure. These should not be generated by the model at all. The model decides what is needed and calls a tool, and ordinary code produces the value.
This is how the AI receptionists we built behind CallGuard AI and CallSetter AI book appointments. The agent does not decide which slots are open. It asks the calendar, gets a real list back, and offers from that list. If the calendar says Tuesday at 3pm is taken, the model never gets the chance to offer it. The same applies to quoting: if a business has not configured a price, the right behavior is to say a team member will confirm it, not to estimate. Voice agent conversation design covers how those boundaries are written into the call flow.
The principle extends well beyond voice. In financial software like Fyuel, the accounting platform we built for the fuel trade, every figure lives in a ledger with agreed definitions. Any AI feature on top of records like that should read and explain the numbers the system already calculated, never recompute or estimate them in prose. Building financial software explains why those definitions matter so much.
A useful test for any AI feature: list every value in its output that someone might act on. For each one, ask whether it comes from a system of record or from the model's wording. Anything in the second group that has to be exact is a design gap.
Pattern 3: Constrain the shape of the output
Free text gives a model unlimited room to invent. Structured output narrows it.
- Fixed fields instead of paragraphs. If an extraction feature returns invoice number, date, supplier and total as separate fields, each can be validated. A paragraph summarizing the invoice cannot.
- Choices from a list. A classifier that must pick one of eight known categories cannot invent a ninth. A ticket router that must choose an existing team cannot route to one that does not exist.
- Validation after generation. Dates must parse, totals must equal the sum of line items, IDs must exist in the database, emails must match a pattern. Output that fails goes back for a retry or to a person, not to the customer.
Most current models support structured output modes that make this straightforward to build. AI document processing shows how this works on invoices and forms, where field-level checks and confidence routing do most of the work.
Pattern 4: Make "I don't know" an acceptable answer
Models are trained to be helpful, which in practice means they lean toward answering. If the product gives them no clean way to decline, they will often produce something rather than nothing.
Build the refusal path deliberately:
- Tell the model what it is allowed to answer from, and that if the answer is not in what it was given, it should say so.
- Give it something useful to do instead. "I don't have that information, but I can have someone call you back today" is a good answer. A bare "I cannot help with that" pushes people to rephrase until the model guesses.
- Design the interface for it. A support assistant that hands off to a person, a voice agent that takes a message, a search tool that says "no matching document" and shows the closest results. Voice AI human handoff covers this for phone calls.
- Measure it. Track how often the feature declines. Too rarely and it is probably guessing. Too often and the underlying content or retrieval needs work.
Pattern 5: Check claims before they are shown or acted on
The final layer is verification: comparing what the model said against the source it was given.
Citations people can click. When an answer quotes or summarizes a document, link to the exact passage. This lets users check quickly, and it makes unsupported claims visible because they have nothing to link to.
Automated grounding checks. A second step, sometimes a smaller model, checks whether each claim in the answer is supported by the retrieved material. It is not perfect, but it catches a meaningful share of drift for very little cost per request.
Stay close to the source. Our own product LectureNotes AI turns lecture recordings into takeaways and outlines. A summary that stays tied to what was actually said in the recording is easier to trust and easier to check than one that adds outside material, so the useful design question is how directly each takeaway can be traced back to the lecture.
A person in the loop where stakes are high. Drafts instead of sends. Suggested replies instead of automatic ones. Proposed changes that need approval. This matters most when the AI can take actions, which is covered in AI agents with write access.
Match the safeguards to the stakes
Not every feature needs all five patterns at full strength. Over-engineering a low-risk feature wastes budget just as under-engineering a high-risk one does.
- Low stakes (brainstorming, first drafts a person will edit, internal summaries): good context and a clear "I don't know" path are usually enough.
- Medium stakes (customer-facing answers, support replies, search over your documentation): add citations, structured output where possible and production monitoring.
- High stakes (prices, bookings, money, legal or medical content, anything written back into a system of record): exact values come from code only, every output is validated, and a person reviews anything the checks are unsure about.
The same product often has features at all three levels, and they should be built differently.
Measure it, because it will drift
A feature that rarely invents answers at launch can start doing so later. The model provider updates a version, someone edits a prompt, a new document contradicts an old one, or customers start asking questions nobody anticipated.
The defense is an evaluation set: real past questions with known correct answers, run against every change before it ships. Evaluating AI features covers how to build one and agree a launch bar. After launch, sample real conversations and outputs regularly, and add every invented answer you find to the set so it cannot quietly return. For voice agents, monitoring AI voice agents covers what to review and how often.
Choosing a stronger model does help, and choosing an AI model covers how to compare them on your own tasks. But a model change should be tested against the evaluation set like any other change, not assumed to be an improvement.
Questions to ask the team building your AI feature
- Where does each fact in the output come from? You want to hear "this record, this document, this tool", not "the model knows".
- Which values are generated by code and which by the model? Anything exact should be in the first group.
- What does the feature do when it does not know? Ask to see it happen.
- How are outputs validated before a customer sees them?
- How will you detect invented answers after launch, and who reviews them?
- What is in the evaluation set, and does it include the questions that tripped up the demo?
The bottom line
Language models will sometimes produce confident, wrong answers, and no vendor can honestly promise otherwise. What a good build does is make those answers rare, visible and harmless: facts supplied rather than remembered, exact values handled by code, outputs constrained and validated, a real path to "I don't know", and claims checked against their source. That is the approach behind the AI systems we have shipped, from CallGuard AI and CallSetter AI reading real calendars instead of guessing availability, to Fortell AI handling intake across more than 100 languages, to our own LectureNotes AI. If you are adding AI to an existing product, plan for this from the first scope document rather than after the first wrong answer.
Building an AI feature that has to be right, not just fluent? Book a demo and walk us through where it reads its facts from. We will show you where invented answers could slip through and how we would close each gap. See our work: CallGuard AI, CallSetter AI, Fortell AI, LectureNotes AI and more.