Case study · AI agents, RAG, evals
An AI call center agent that knows when to hand off
An agent that answers routine inbound calls for a home-services company: it answers questions from the company's own documents, books and moves appointments in a real scheduling database, and hands everything else to a person with a ticket they can act on.
- 96%right source in the top 3 (66 of 69)
- 81%of allowed calls resolved (178 of 221)
- 0false containments, out of 139
- 96 / 96red-team attacks passed
The question first
The brief asked for 60% containment. Before writing any agent code, I worked out why customers call. I drew the reason categories from the data and signed off on them, then had two blind AI labelers from different model families classify all 420 sample calls. They agreed on every call, a spot check of 20 matched the transcripts, and the top five reasons covered 90% of calls.
That analysis changed the goal. Many calls must go to a person no matter how good the agent is: complaints, safety issues, and requests that need tools the company doesn't have. Even a perfect agent tops out at 61% of the test calls. So the real question became: of the calls it's allowed to handle, how many does it resolve, and does it ever keep a call it shouldn't?
What I built
- A supervisor and two workers. A fast model routes each message to a knowledge-base worker or a scheduling worker.
- A code gate before any model runs. Safety and crisis phrases, requests to talk to a person, and limits on the public app are handled in plain code.
- Code decides what matters. Who the caller is, whether a booking matches an option they clearly agreed to, fee consent and every handoff ticket are decided in code, not by the model.
- Every reply is checked before it's sent. Every price, date, time and phone number in a reply must appear in the sources the agent retrieved.
- Production basics. A staff dashboard with containment, handoffs, response times and spend against a $15/day cap; automatic flags for bad answers; hourly health checks and daily scripted test calls.
How I measured it
I kept a development set of 60 calls for fixing problems and evaluated on the other 360, including 150 held-out calls. An offline judge checked every claim in every answer against what the agent had retrieved, and a red-team suite ran 30 attacks in several disguises with a bar of 100%.
I also compared models. Running the cheaper model as the agent's brain cut cost per conversation from 2.5¢ to 0.4¢, but it kept two calls that should have gone to a person. For a customer-facing agent, that settled it.
Results
- 49% containment of all 360 test calls, against a 61% ceiling, and 81% of the calls it's allowed to handle.
- Zero false containments: it never kept a call that had to go to a person.
- Every remaining call was handed off, 87% of them with the exact reason.
- 84% of answers fully grounded in retrieved sources, and 96 of 96 attacks blocked, with no data leaks.
What's still open
The containment ceiling is set by the company's operational gaps, not the agent, so I wrote up those gaps as the client's lever. Scheduling replies get the number check but not yet the citation check, chat state is lost on restart, and voice is designed but not built.