How we stop an AI agent from making things up in production

When an agent gives a bad answer, the instinct is to blame the wording and rewrite the prompt, or to buy a bigger model. Prompt quality does matter, and it moves the numbers, but it does not change the shape of the problem. The failure rate depends heavily on what you ask the model to do. Industry benchmarks through 2025 and 2026 put open-ended generation, where the model writes free text from its own memory, at hallucination rates anywhere from 40 to 80 percent. Narrow the task to a closed question answered from a supplied document, and the rate drops toward 10 to 20 percent. Grounded summarization, where the model only compresses text you hand it, has been measured under 2 percent.

Read that as an instruction, not trivia. The single biggest lever is not the prompt, it is the job. The more you let the model invent from memory, the more it invents. The more you force it to work from real data in front of it and produce a checkable result, the safer it gets. Everything below is built on that one idea: shrink the space where the model is free to make things up, and put a check on everything it produces.

Grounding means the agent does not answer from what it vaguely remembers from training. It answers from real records: your database, your order history, your documents, the actual email thread. For anything factual, we make the agent fetch the source first and answer from that, and where it matters we make it cite which record it used so a person can trace the claim back. If the answer is a number, that number comes from a live lookup, not from the model's sense of what looks about right.

This matters because the domains where models fail hardest are exactly the ones people want to automate. Benchmarks report medical question rates above 60 percent without grounding, and legal research queries producing false citations somewhere between 58 and 88 percent when the model is left to recall case law from memory. The pattern behind those numbers is consistent: when the passage that actually answers the question is never put in front of the model, no model, however capable, can answer correctly. Grounding is not a nice extra. It is the difference between a system that reports your reality and one that produces plausible fiction about it.

Grounding reduces how often the agent is wrong. Validation catches the times it still is. Before any output is allowed to cause an action, it passes through checks that do not involve the model at all. These are ordinary rules, the kind an experienced clerk would apply on instinct, written down and run every single time. If a check fails, the action does not happen. The item goes to a person instead.

• Arithmetic: line items sum to the stated total, a refund does not exceed the original charge, percentages land in a sane range.
• Format: an email is a valid email, a date is a real date, an invoice number matches the pattern your system uses.
• Ranges and rules: a discount stays inside policy, a quantity is positive, a payment amount falls under the limit that is allowed to run without a person.
• Existence: the customer, the order, the SKU actually exist in your system. This alone kills a large class of failures, because a fabricated ID simply fails the lookup.

A useful trick that makes this stronger is structured output. Instead of asking the model for a paragraph, we ask it to fill specific typed fields: amount, currency, account, action. Filling known slots gives the model far less room to wander than free prose, and it makes the result trivial to validate field by field. It turns a text-generation call into something much closer to a typed function call, which is a thing a computer can check.

The most dangerous behavior is a model that guesses confidently rather than admitting it is unsure. The OpenAI research named this directly: models are trained in a way that punishes humility, so they learn to always produce an answer. A 2026 Nature paper made the same point, that evaluating models mainly on accuracy actively incentivizes hallucination, because saying "I do not know" earns the same zero as a wrong guess. We design against that incentive. In our systems, "I am not sure" is a legitimate, expected output, and it routes the item to a human instead of forcing a decision.

One practical way to surface uncertainty is self-consistency. For a decision that matters, you can have the model answer the same question more than once, or from slightly different angles, and compare. When the answers agree, that is a weak signal of confidence. When they disagree, that disagreement is the tell that the ground is shaky, and the item gets escalated rather than actioned. It is not a guarantee, and we do not treat it as one, but a cheap disagreement check catches a meaningful slice of the confident-but-wrong cases before they do harm.

For anything sensitive, money moving, a customer being contacted, a change to the books, a person approves before it happens. This is not a temporary training-wheels phase we are embarrassed about. It is the default, and it stays the default until an agent has earned trust on a specific task by being right on your real data, repeatedly, with a paper trail to prove it. Autonomy is granted narrowly and slowly, one action type at a time, never handed over wholesale.

The human queue is where all of this comes together. When the agent is confident and every validation passes, the work flows. When grounding comes up empty, a check fails, the model signals uncertainty, or the action crosses a sensitivity threshold, the item lands in a queue for a person to approve, correct, or reject. The person is not re-doing the work. They are ruling on the small fraction that the system itself flagged as not safe to run alone. That is a very different and much lighter job than doing everything by hand, and it means the failures you care about get a human set of eyes by design, not by luck.

There is a temptation to hand an agent a big toolbox and let it figure out the rest. The evidence says do not. Work around the Berkeley Function Calling Leaderboard and related testing through 2025 and 2026 found that tool-calling accuracy drops as the number of available tools grows. More tools mean more documentation to parse, more similar-looking options to confuse, and more chances to call the wrong one, pass the wrong parameters, or invent a value. Models have been observed producing tool names that do not exist and arguments that were never real.

This is why we run one workflow at a time, with the smallest set of tools that job needs, rather than a general-purpose agent with the keys to everything. A narrow agent with four well-chosen actions is more reliable and far easier to reason about than a sprawling one with forty. It also makes every failure easier to trace, because there are only so many things the agent could have done. Scope is a safety feature, not a limitation we apologize for.

Every action the agent takes is logged: what it saw, what it decided, which source it used, what checks it ran, and what it did or handed off. This audit trail is what makes the system honest. When something goes wrong, you do not argue about what the AI "probably" did. You read the record. It is also how a person can trace any single decision back to the exact data behind it, which matters a great deal when the thing being automated touches money or customers.

Before we let an agent run a workflow for real, we prove accuracy on your own data, not on a vendor's benchmark. We build a paid prototype, run it against your actual records, and measure it against a baseline: how the task is done today, and how often that is right. We do not quote borrowed accuracy numbers as if they were yours, because a model's score on a public test tells you very little about how it will do on your invoices, your customers, and your edge cases. The only accuracy figure worth trusting is the one measured on your work.

That measurement continues after launch. Evaluation is not a one-time gate. We keep scoring the agent's outputs against known-good answers and against the human queue's corrections, so drift and new failure modes show up as numbers rather than as a surprise in your accounts three weeks later.

No one can guarantee zero mistakes, and anyone who tells you otherwise is selling something. The math is not on their side. The OpenAI work showed hallucination is bounded below by a model's underlying error rate on the task, which is a formal way of saying some rate of being wrong is baked in. Even the strongest models in 2026 leaderboards cluster around 10 to 20 percent on hard factual tasks, and one prominent reasoning model was measured hallucinating on roughly a third of a person-focused benchmark. These systems are useful and they are also fallible, both at once.

So we do not aim for a perfect agent. We aim for a system where the mistakes it does make are caught by grounding, stopped by validation, flagged by an uncertain answer, or held at the human queue, and where anything that still slips through is sitting in a log you can read. The goal is not an AI that never errs. It is an operation where errors surface as a flagged item for a person to handle, not as a quiet wrong number in your books. That is a bar we can actually hold ourselves to.