Should we use a frontier AI model or a smaller, cheaper one?

Both, in different places. Most production AI work is made of narrow tasks with a defined shape: classify this message, extract these six fields, decide whether this call needs a human, rewrite this in the house tone. On work like that a small, fast model is frequently indistinguishable from the largest one in the output your users actually see, at a fraction of the cost and a fraction of the response time. The frontier tier earns its price when the task is genuinely open-ended, when the input is unpredictable, or when quality on the hard cases is what the product gets judged on. The mistake we see most often is routing an entire product through the largest available model because it was the safe choice during the prototype, which quietly sets a cost curve that scales with usage rather than with revenue. Break the feature into its narrow parts and its genuinely hard parts, send the easy majority to the cheap model, and reserve the expensive one for the cases that need it. That single split is usually the largest cost reduction available in an AI build, and it makes the system easier to evaluate as well, because narrow tasks have checkable answers.

Do we need to fine-tune a model for our product?

Usually not, and it is worth resisting because fine-tuning is proposed far more often than it is needed. Most of what teams want from fine-tuning, domain vocabulary, house tone, consistent output structure, knowledge of their own content, is solved better by giving the model the right context at request time: cleaner retrieval, a sharper prompt, a defined output schema. That path is faster to build, cheaper to iterate on and easy to change when requirements move. Fine-tuning also creates lock-in, since a tuned model ties you to one provider and one model version, and unwinding it when that version is deprecated or something better arrives is expensive. The cases where it does pay off are narrow and specific: a highly specialized output format or domain language that prompting cannot reach reliably, or a high-volume task where a smaller tuned model replaces a much more expensive general one and the savings clearly cover the work. Test the prompt-and-retrieval version against a real evaluation set first. If that clears your quality bar, and it usually does, you have saved yourself a dependency.

Is it safer to self-host an open-source model for sensitive data?

It is more controlled, which is not automatically the same as safer. Self-hosting an open-weight model means inference happens on infrastructure you own, which is the right answer when a compliance requirement, a data-residency rule or a contract genuinely rules out sending data to a third party. What comes with it is an infrastructure project: sizing GPUs, managing capacity, handling failover, patching and owning the upgrade path, plus the security of that environment, which is now your responsibility rather than a provider's. A poorly run self-hosted deployment is not safer than a well-configured hosted one under enterprise terms that cover retention and training use. Before assuming self-hosting is required, check two cheaper options. First, whether your provider offers deployment in your required region with contractual terms your compliance owner accepts, which covers a large share of cases. Second, whether the sensitive fields need to reach the model at all, since stripping or tokenizing identifiers before the request often removes the problem entirely. Settle this in writing before the build starts, because it is a design input rather than a review step.

Which AI Model Should Your Product Use? Choosing Without Chasing Benchmarks

TL;DR: The model is one replaceable component in an AI product, and it is rarely the reason the product works or fails. Choose it against five constraints: how fast the answer has to arrive, where the data is allowed to go, how narrow the task is, what it costs at real volume, and which languages and inputs it has to handle. Public benchmarks will not settle it, and a set of thirty real examples from your own workload will. Then build so the model can be swapped, because the one you pick today is not the one you will be running in a year.

Almost every AI project we scope arrives with the same opening question: which model should we use? It gets asked as though it were the architectural decision, and it is usually one of the least consequential ones on the list. Teams spend weeks on it and then ship something that fails for reasons the model was never going to fix.

What decides whether an AI feature is good is mostly everything around the model: which context it gets handed, how that context is retrieved, what it is allowed to do, what happens when it is unsure, and whether anyone is measuring the output after launch. We wrote about the retrieval half of that in building an internal AI knowledge base, and the same lesson applies here. A strong model on weak context loses to a modest model on clean, well-scoped context every time.

The choice still matters: it sets your cost curve, your latency ceiling, where your data goes and how easily you can change your mind later. Here is how we work through it.

The three kinds of model you are choosing between

Model names change every few months. The categories do not, and the categories are what you are actually deciding between.

Frontier hosted models

The largest general-purpose models from the major labs, accessed over an API. They handle open-ended reasoning, messy instructions and unusual inputs better than anything else available. You pay per unit of usage, you inherit the provider's uptime and rate limits, and your data leaves your infrastructure under whatever terms you agreed to.

Right when the task is genuinely open-ended, or when quality on the hard cases is what the product gets judged on.

Small, fast hosted models

Smaller models from the same providers, sold on speed and price rather than raw capability. For narrow, well-defined work they are frequently indistinguishable from the frontier tier in the output your users see, at a fraction of the cost and a fraction of the response time.

Right when the task has a shape: classify this message, extract these six fields, decide whether this call needs a human, rewrite this in the house tone. Most production AI work is this kind of work, and teams routinely overpay by running all of it through the largest model available.

Open-weight models you run yourself

Models whose weights you can download and host, on your own cloud account or your own hardware. You get control over where inference happens, no per-token bill, and no dependency on a provider's roadmap or pricing.

You also get an infrastructure project. Someone has to size GPUs, manage capacity, handle failover and own the upgrade path, and that ongoing cost is easy to underestimate at scoping time. Right when a compliance requirement, a data-residency rule or a high steady volume makes the trade worth it. Wrong as a default, and wrong as a way to save money on a product that does not have traffic yet.

The five constraints that actually decide it

1. How fast does the answer have to arrive?

This is the constraint that eliminates options fastest, and it is set by the interface rather than by the technology.

A voice agent is the strict end. In a phone conversation the model has to respond inside the length of a natural pause, because anything longer reads as dead air and the caller starts talking over it or assumes the line dropped. In the voice systems we build for CallGuard AI and CallSetter AI, that budget also has to absorb speech recognition, any lookup the agent performs and speech synthesis, so the model gets only a slice of it. That pushes hard toward smaller, faster models for the conversational turn, with anything slower moved off the call.

A chat interface is more forgiving, because streamed text starts arriving immediately. A job that summarizes yesterday's tickets overnight has no meaningful latency constraint at all and should use whichever model gives the best output per dollar. Decide which of those three you are building before you compare anything.

2. Where is the data allowed to go?

Ask this early, in writing, with whoever owns the answer. Health records, financial records, government benefit applications and anything covered by a customer contract come with rules about where processing may happen and who may retain it.

Most providers offer enterprise terms covering retention and training use, and for a large share of businesses that is sufficient. When it is not, the honest answers are a self-hosted open-weight model, a provider deployment inside your own cloud region, or stripping the sensitive fields before the model ever sees them. What does not work is discovering the constraint after the build. We went through the same reasoning in building financial software: the compliance requirement is a design input, not a review step.

3. How narrow is the task?

Write down what the model is being asked to do in one sentence. If that sentence contains a fixed list of outputs, or a fixed set of fields, you have a narrow task and you should be trying the cheapest model first. If it contains "and then decide what to do next", you have an open-ended task and the extra capability is worth paying for.

Products are usually a mix. In LectureNotes AI, our note-taking product, transcription, outline structuring and summary quality are not the same job and do not deserve the same model. Splitting a feature into its narrow parts and its genuinely hard parts is usually the largest cost reduction available.

4. What does it cost at real volume?

Model cost scales with usage, not with seats. Do the arithmetic at the volume you expect a year in rather than at demo volume, and do it per unit of business value: cost per call handled, per document processed, per active user per month. If that number sits uncomfortably close to what you charge, the fix is architectural rather than a matter of negotiating rates. Cache what repeats, cut the context down to what is actually needed, route the easy majority to a small model and reserve the expensive one for the cases that need it.

5. What languages and inputs must it handle?

Language coverage varies more than the marketing suggests, and a model that is excellent in English can be noticeably stiff in the language your users actually speak. The AI voice and SMS systems we build for Fortell support intake in more than a hundred languages for Community Action Agencies, and that requirement rules out options the English-only version of the same product would happily use. If you are going multilingual, the multilingual voice AI trade-offs are worth reading first.

Benchmarks will not decide this, your own eval set will

Public leaderboards measure general capability on shared tasks. You are building one narrow thing for one set of users, and a leaderboard ranking predicts your outcome poorly.

The replacement is unglamorous and takes about a day. Collect thirty to fifty real examples from the workload: actual customer messages, actual documents, actual call transcripts, including the awkward ones. Write down what a good response looks like for each. Run your candidate models against that set and read the outputs yourself.

That set keeps paying off long after the decision, because it becomes the regression suite you run when you change a prompt, when a provider updates a model underneath you, and when you evaluate a switch. Without it you cannot tell an improvement from a regression, and you are choosing on impressions. It is the same discipline as watching the thing in production, which we covered in monitoring an AI voice agent after launch.

Build so you can switch

Assume the model will change. Providers deprecate versions, prices move, and something better arrives roughly as often as you finish a build.

Keep every model call behind one internal interface rather than scattered across the codebase. Keep prompts and model selection in server-side configuration, so changing either is an edit rather than a release. Avoid hard dependencies on one vendor's proprietary orchestration unless it is earning its keep. None of that is expensive at the start and all of it is expensive to retrofit, which is the same portability argument we make about owning your source and your deployment path in software ownership and handover.

Questions to ask before you commission an AI build

What is the latency budget, and which interface sets it? If nobody can state it, the model has not been chosen against anything.
What data does the model see, and where is it processed? Get this in writing from whoever owns the compliance answer.
Which parts of this are narrow tasks, and are they running on the cheapest model that works? One model for everything is a cost decision made by accident.
What is the cost per call, per document or per user at next year's volume?
What does the evaluation set look like, and who wrote the expected answers? Real examples, or nothing.
How long does swapping the model take? If the answer is more than a config change and a test run, the build is more coupled than it should be.

The bottom line

There is no best model, only the one that fits your latency budget, your data rules, the shape of your task, your volume economics and your language coverage. Get those five written down and the field narrows to a couple of candidates before anyone opens a comparison chart. Decide between them on your own examples, then keep the decision cheap to revisit. That is a large part of why our builds ship in days rather than quarters, as set out in the AI-native studio playbook.

Choosing the stack behind an AI feature and want a straight answer rather than a leaderboard? Book a demo and we will work through the latency, data and volume constraints with you, including telling you when a small model or plain software gets you there faster. See our AI work: voice and intake systems for CallGuard AI, CallSetter AI and Fortell, plus our own products LectureNotes AI and Lifemaxxing AI.