TL;DR: The AI model behind your feature will be replaced. Providers retire older versions on a published schedule, update models under the same name, change prices and release better options, and none of that waits for your roadmap. The teams that handle this calmly did three things before launch: they kept every model call behind one internal interface, they built a test set of real examples with expected results, and they named someone responsible for watching provider notices. With those in place a forced migration is a short, measured project. Without them it is a scramble against a deadline someone else set. If you are commissioning an AI build, ask how a model swap works before you ask which model it uses.
Most buyers think about the model once, at the start, when they choose it. That is the right time to choose, and the wrong time to stop thinking. A traditional software dependency can sit on an old version for years. A hosted AI model usually cannot, because the provider decides when it goes away.
This guide covers what actually changes, what breaks when it does, how to prepare during the build, and how to run an upgrade without your users noticing anything except, ideally, better results.
What actually changes, and how often
Four kinds of change reach a live AI feature, and they need different responses.
Scheduled retirement. Providers announce that a specific model version will stop being served on a set date. After that date, calls to it fail or are redirected. You get notice, but you do not get a vote.
Silent updates under the same name. Some model names point to "the latest version" rather than a fixed snapshot. When the provider moves that pointer, your feature starts running on a different model without any change in your code. Behavior can shift in small ways: tone, length, formatting, how strictly instructions are followed.
Price and limit changes. A new model may be cheaper per call and use more tokens per answer, or the reverse. Rate limits and context sizes change too. A swap that looks free on the price sheet can change your monthly bill, which is why we treat cost as a test result rather than an assumption, as covered in AI cost control.
Better options arriving. Not every change is forced. A newer or smaller model may do your task as well for less money or with lower latency. These are the upgrades you choose, and they are only cheap to take if the forced ones are cheap too.
What breaks when the model changes
A new model rarely fails outright. It fails in the details your product quietly depends on.
- Output format. Code that expects a specific JSON shape, a fixed list of categories or a particular field order can break when a model starts adding commentary or renaming keys.
- Instruction following. Prompts tuned over months to one model's habits can be read differently by the next. Rules that were always obeyed may become occasional.
- Length and tone. Answers get longer or shorter, more formal or more casual. In a voice agent, longer answers also mean slower turns, which we discuss in voice AI latency.
- Refusals and caution. A newer model may decline requests the old one handled, or handle requests the old one refused.
- Tool calling. Agents that call your systems depend on the model choosing the right tool with the right arguments. Small changes here have real consequences when the agent can write data, as covered in AI agents with write access.
- Accuracy on your edge cases. A model that scores better on public benchmarks can still do worse on the awkward ten percent of your real traffic.
None of this shows up in a quick demo. It shows up in support tickets a week later, unless you measure it first.
Prepare during the build, not after the notice
Most of the cost of a model migration is decided long before the provider emails you. These are the things we put in place during the initial build.
One interface for every model call
Every call to a model should go through one internal module, not be scattered across the codebase. That module owns the provider client, retries, timeouts, logging and the choice of model. Changing models then means changing one place. We make the same case for portability in choosing an AI model and software ownership and handover.
Pin versions and keep configuration server-side
Use fixed model versions in production rather than names that float to the latest release, so changes happen when you decide. Keep the model name, prompts and key settings in server-side configuration, so a swap or rollback is a configuration change rather than an app store release. For mobile products this matters doubly, since a release can take days to reach users, as we explain in the app store release process.
Validate outputs in code
Do not trust that the model returns what you asked for. Check the structure, check that categories are from the allowed list, check that required fields exist, and decide what happens when a check fails: retry, fall back or route to a person. Validation catches format drift on the first bad response instead of the hundredth.
Build the test set
This is the single most valuable asset in any migration. Collect real examples from your workload, including the difficult ones, and record what a good result looks like for each. Run it against the current model to get a baseline. When a replacement is needed, run the same set against the candidate and compare. The method is laid out in evaluating AI features, and it is the same suite you will use for every prompt change afterward.
Log enough to compare
Keep the inputs, outputs, model version, latency and cost for each call, with sensible privacy controls. When something changes, you want to answer "is this different from last month?" with data rather than impressions. The privacy side of that logging is covered in AI feature security.
Running the upgrade
When a retirement notice arrives, or a better model appears, the process is short if the groundwork exists.
Step 1: Read the notice and set the real deadline
Note the retirement date, which versions are affected and what the provider recommends as a replacement. Set your own deadline well before theirs, so a failed first attempt does not become an outage.
Step 2: Shortlist candidates
Usually two or three: the provider's suggested successor, a smaller or cheaper option, and sometimes a model from another provider. Check each against your constraints on data processing, regions and latency before testing quality.
Step 3: Run the test set
Compare each candidate to the baseline on accuracy, format compliance, latency and cost per task. Read a sample of outputs yourself, especially the failures. Expect to adjust prompts for the new model, and rerun after each adjustment.
Step 4: Roll out gradually
Send a small share of real traffic to the new model first, or run it in shadow mode where it processes requests but its answers are only logged. Watch the same signals you watch in normal operation. Our guide to monitoring an AI voice agent after launch lists those signals for voice, and most apply to any AI feature.
Step 5: Keep the rollback path open
Until the old model actually retires, keep the ability to switch back with a configuration change. After it retires, keep the previous prompt and configuration in version control so you understand what changed if a problem surfaces later.
When the migration is a chance to improve
Forced upgrades are also the cheapest moment to revisit decisions. You are already running the test set and reading outputs, so it costs little to ask:
- Does part of this task now fit a smaller, faster model?
- Would retrieval do better than the long instructions we pasted into the prompt? We compare those approaches in fine-tuning vs RAG.
- Could a narrow step move on-device, as discussed in on-device AI vs cloud AI?
- Are there failure cases in the test set we have tolerated for too long?
A well-run migration often leaves the feature cheaper and more accurate than before, not just still working.
Who owns this after launch
Model changes are an operational responsibility, not a one-off project. Someone has to read provider announcements, track which features use which model versions and schedule upgrades before the deadlines. If your studio or vendor runs the feature for you, that should be written into the maintenance agreement, with a clear statement of how much notice they will give you and how a migration is billed. Our view on what ongoing care should include is in software maintenance after launch.
If you are inheriting an AI feature from someone else, check how it calls the model before anything else. Scattered calls, floating model names and no test set are the most common reasons a routine retirement turns into a rewrite, and they are worth fixing early. We cover that kind of assessment in technical due diligence.
Where we fit
We build AI features as replaceable parts from the first version, because the ground under them keeps moving. Our voice work for CallGuard AI, CallSetter AI and Fortell AI runs conversations, bookings and intake where a behavior change shows up on live calls, and Fortell's intake across 100+ languages means a model change has to be checked in more than one language, not just in English. Our own products, LectureNotes AI and Lifemaxxing AI, depend on summaries and personalized plans that users notice when they shift. The same discipline applies in each: one interface, pinned versions, a real test set and a staged rollout.
Questions to ask before you commission an AI build
- How many places in the code call a model directly?
- Are production model versions pinned, and where is that setting stored?
- Can we switch models or roll back without shipping a new app release?
- What does the test set contain, who wrote the expected results and can we see the baseline?
- How are model outputs validated before they reach users or other systems?
- Who watches provider retirement notices, and how much warning will we get?
- How is a forced migration scoped and billed under the maintenance agreement?
- If we move to another provider, what has to change besides configuration?
Facing a model retirement notice, or want an AI feature that can change models without a rewrite? Book a demo and we will review how your feature calls its model, build the test set and plan a migration that users never notice. See our AI work: CallGuard AI, CallSetter AI, Fortell AI, LectureNotes AI and Lifemaxxing AI, shipped in days, not months.