It is now easy to build an agent that answers questions well in a meeting. It is still hard to build one you would let touch a customer record unsupervised. The difference is not model quality, it is the scaffolding around the model.
Separate answering from acting
An agent that retrieves and explains is a low-risk product. An agent that updates a CRM, issues a refund, or changes an order is a different category of thing, because a mistake persists.
Treating these as one feature is the most common design error we see. Answering can be fairly open-ended; acting should be a small, explicit, well-tested set of operations with clear preconditions.
Ground answers in your data, and cite them
Agents that answer from the model's general knowledge are confidently wrong about your business in ways that are hard to detect. Retrieval against your own documents and records fixes most of this.
Showing the source alongside the answer matters as much as the retrieval itself. It lets a user check the reasoning in two seconds instead of trusting it, and it turns "the AI said" into "this document said."
Confidence thresholds and handoff
Every useful agent needs a defined answer to "what happens when I am not sure?" and the answer cannot be "answer anyway." Below a threshold, the agent should hand off to a person, with the conversation and its own partial findings attached so the human does not start from nothing.
This is the same principle as ambiguity protection in our OCR matching work, and for the same reason: a system that escalates when uncertain is trustworthy, and a system that always produces an answer is not.
- Define the confidence floor before launch, not after an incident
- Hand off with context, never with a dead end
- Log every escalation, the pattern tells you what to build next
- Make handoff visible to the user, so they know a person is coming
Budgets, logging, and a kill switch
Agents cost money per call and can loop. In the compliance platform we worked on, administrators could set usage budgets, watch spend, configure provider fallbacks, and disable AI features outright with a kill switch.
That last control sounds pessimistic but it is what makes the feature deployable. Anyone accountable for a system needs to know they can stop it without a code change.
Constrain the domain deliberately
The strongest agents in production are narrow. An agent for order status, or invoice questions, or checking who has responded to an invitation, can be evaluated properly. You can write down what a correct answer looks like and test against it.
A general assistant for your whole business cannot be evaluated, which means you never find out whether it got worse.