What happens when the AI model changes underneath your automation

When you build an automation on a hosted model from OpenAI, Anthropic, or Google, you are renting behavior, not owning it. The provider can update the model's weights, adjust its safety tuning, change its pricing, or schedule it for shutdown. You get a version string and an API, and behind that string the actual thing answering your prompts can move. This is a reasonable trade. You get frontier capability without training anything yourself. But it means the stability of your automation depends on decisions made by a company whose roadmap you do not see.

The pace here is faster than most owners expect. One widely shared tally noted that Anthropic issued deprecation notices for eight Claude models inside a single twelve-month stretch. OpenAI retired GPT-4o, GPT-4.1, GPT-4.1 mini, and o4-mini from ChatGPT on February 13, 2026, and API calls to several of those older models started returning errors within days. Google retired a set of Gemini 2.0 model IDs on a June 1, 2026 shutdown date. None of this is a scandal. It is the ordinary lifecycle of a fast-moving field. It just means a model you picked eighteen months ago is unlikely to still be the current default, and may not exist at all.

The failure people plan for least is the one that makes no noise. A model gets a provider-side update, keeps the same name, and starts behaving differently. Nothing errors. Nothing alerts. The prompt you validated last quarter now returns answers that are a little more cautious, or formats output slightly differently, or refuses a request it used to handle. In drift-monitoring terms, this is model drift: a provider-side weight change that shifts behavior without notifying you, as distinct from prompt drift, which is a change you introduced yourself.

There is a well-documented example. A 2023 study from Stanford and UC Berkeley, titled "How Is ChatGPT's Behavior Changing over Time?" by Lingjiao Chen, Matei Zaharia, and James Zou, measured the same tasks against GPT-4 in March and again in June. On one task, identifying whether a number is prime, GPT-4's accuracy dropped from 97.6 percent in March to 2.4 percent in June on the same questions. The chain-of-thought prompting that helped in March stopped helping. The researchers were running identical prompts. The model underneath had shifted. Whether that particular swing was pure capability loss or a change in formatting and instruction-following was debated at the time, but the headline point held: the thing behind the name moved, and anyone depending on the old behavior would have felt it.

For a small business, drift rarely shows up as a dramatic accuracy cliff. It shows up as an agent that starts miscategorizing a certain kind of expense, or a support reply that grows a stiff new disclaimer, or an extraction step that quietly changes how it writes dates. Small, plausible, and easy to miss until it has been wrong a hundred times.

The louder risk is retirement. A model is not just updated, it is switched off. The vendors handle this more responsibly than they used to, and each now publishes a lifecycle policy, but the details differ and the notice windows are shorter than you might hope.

• OpenAI: deprecation means the model is scheduled for shutdown, after which API calls fail. Their stated minimums are roughly 6 months of notice for generally available models, about 3 months for specialized variants, and as little as 2 weeks for preview models. Preview and dated-preview models are explicitly not meant for business-critical work unless you can migrate fast.
• Anthropic: models move through Active, then Legacy (no more updates), then Deprecated (still works, replacement named, retirement date set), then Retired (gone). Their commitment is at least 60 days notice before retirement for publicly released models, with email and documentation notices to affected customers.
• Google (Gemini): models ship as stable, preview, latest, or experimental. Stable versions are the ones meant for production and mostly do not change. Preview models can be deprecated with as little as 2 weeks notice. Published shutdown dates are the earliest possible retirement, not a promise the model lasts that long.

Read those together and a pattern emerges. If you are on a stable, generally available model, you likely have months of warning. If you are on a preview or experimental model, which is often the newest and most tempting one, you may have two weeks. A worthwhile detail: when Anthropic retired Claude Opus 3, it kept the model available to paid users on its own product and by request on the API, rather than deleting it outright. That is a sign the industry is maturing, but it is a courtesy, not a guarantee you should build on.

The reflex when a model is retired is to point the automation at the newest release and move on. Sometimes that works. Often it introduces a fresh round of the exact drift problem, because the new model is genuinely different. It may be better on average and still worse on your specific task. It may format output in a new way, follow instructions differently, or have different refusal boundaries. Newer does not mean identical, and for an automation that feeds your books or your customers, identical is what you actually want on the day of a swap.

There is also a cost dimension people forget. Vendors re-price models over their lifecycle, and the newest or most capable model is frequently the most expensive per token. A migration forced by retirement is also a moment where your running cost can jump if nobody checks. So the honest version of "just use the newest model" is: evaluate the new model against your real work, confirm it holds up, watch the new price, and only then switch. That evaluation step is the whole game, and it is what most quick builds skip.

Two engineering practices do most of the protective work, and they pull in opposite directions in a useful way. The first is version pinning. Instead of pointing your automation at a floating alias like "gpt-4o", which silently updates to whatever the latest version is, you pin a dated snapshot like "gpt-4o-2024-08-06". A pinned snapshot is stable: the provider commits not to change its behavior underneath you. This is how you stop silent drift from reaching you by accident. You choose when to move, rather than being moved.

The second practice sounds contradictory but is not: keep the model swappable in the architecture. The specific model should be one configurable setting, not something hard-wired through the middle of your automation. When the provider gives a retirement date, or a newer model proves better and cheaper on your cases, swapping should be a config change and a test run, not a rebuild. Pinning gives you stability day to day. A swappable design gives you a clean, fast move when you decide, or the vendor forces you, to change. You want both. Pinned so nothing shifts by surprise, swappable so a required shift is routine rather than a project.

Pinning and swappability only matter if you can tell whether a new or updated model still does your job. That is what a regression test set is for. In practice it is a collection of your real cases, the actual invoices, tickets, emails, or records your automation handles, each paired with the correct outcome you already agreed on. In the LLM testing world this is sometimes called a canary or gold set: a versioned, dated set of prompts with recorded expected outputs, re-run whenever the model, prompt, or tools change, and compared against the last known-good baseline.

The discipline is simple to state and easy to neglect. Every time a model changes, whether you initiated the change or the vendor did, you re-run the set and compare. If the new model matches the baseline on your cases, you move with confidence. If forty of two hundred cases now come out differently, you have found the problem in a test, before a single real customer or real ledger entry was touched. The reason to build this set from your own data, and not borrowed benchmarks, is that a model can score well on public tests and still miss the specific way your business writes purchase orders. Accuracy that matters is accuracy on your work, which is why we prove it on your real cases with a paid prototype rather than quoting someone else's numbers.

Regression tests run before a change ships. Monitoring runs after, on live traffic, and it exists to catch the drift that testing could not have predicted, including the silent provider-side update on a model you thought was pinned. The two are complementary: testing is what you do before deploy, observability is what you do after. Together they close the gap where a model quietly shifts between your scheduled checks.

In a human-in-the-loop setup, monitoring has real teeth because a person still approves anything sensitive: money moving, a customer being contacted, the books being touched. If the agent's behavior drifts, the approver sees odd output before it becomes an action, and every action is logged in an audit trail you can review. On a live production system we operate, an AI agent works as the monitoring and operations engineer around the clock, watching for exactly this kind of shift, so drift is detected and corrected rather than discovered weeks later in a reconciliation. The goal is boring: catch the change while it is still a flag on a screen, not an entry in your accounts.

Retirement is the most predictable of these events, because it comes with a date. That makes it manageable if someone is tracking the dates. The work is straightforward: watch the vendor deprecation notices for every model your automations use, treat the published shutdown date as the deadline (remembering Google's dates are the earliest possible, and preview models can go with two weeks notice), choose the replacement, run it against your regression set, confirm cost and behavior, and switch well before the cutoff. Done this way a retirement is a scheduled maintenance task. Done reactively it is an outage on the vendor's timetable.

The failure mode is not knowing. An owner running an automation built once and never maintained often finds out a model was retired when the automation starts failing, or worse, when it starts silently doing nothing. Because the retirement dates are public and emailed to the account holder, this is entirely avoidable. It just requires that someone is reading those notices and acting on them on your behalf, which is a job, not a one-time deliverable.

Everything above is ongoing work: pinning versions and choosing when to move, keeping the model swappable, maintaining a regression set from your real cases and re-running it on every change, monitoring live output for drift, and handling retirements before their deadlines. This is why a maintenance layer exists separately from the initial build. The build makes the automation work today. The maintenance keeps it working while the models underneath it are updated, re-priced, and retired around it.

At Passcut this is the Care retainer: the ongoing part where we keep your workflow pinned, tested, and current, watch the vendors' deprecation schedules for the models you depend on, and migrate ahead of deadlines rather than after failures. You own everything (the code, the config, the keys, the data, in your own accounts under no-training terms), and Care is the standing arrangement that keeps it stable. If you already run an automation and have never touched the model since it launched, that is worth a look now, and the free workflow audit is a low-commitment way to see where you stand before a retirement date makes the decision for you.