Containment rate — the share of conversations that never reached a human — is the headline chatbot metric almost everywhere, and it treats a customer who gave up in frustration exactly the same as one whose problem was solved. Both avoided an agent. That makes it maximisable by being unhelpful, which is a bad property for a target. The honest alternative needs no survey: measure how many people come back within a few days, and price a wrong answer differently from an escalation, because they cost very different amounts.
Nearly every chatbot business case is built on deflection. So many conversations a month, this proportion handled without an agent, that much saved. It is a clean argument and the number at its centre is doing something it cannot do.
Three outcomes, counted as two
A conversation with a bot ends one of three ways. The customer's problem gets solved. The customer reaches a human. Or the customer gives up — closes the window, calls a different number, posts publicly, or quietly stops being a customer.
Containment sees only whether an agent was involved. So it counts the first and third identically. The one unambiguous success and the one unambiguous failure land in the same bucket, and the only outcome it penalises is the honest escalation in the middle.
Why this is worse than an imprecise metric
Plenty of measures are approximate and still useful. This one is different, because the cheapest way to improve it is to do the wrong thing.
Want a higher containment rate by Friday? Bury the route to a human. Remove the "talk to someone" button from the first reply. Make the bot answer with more confidence and fewer qualifications so conversations end sooner. Every one of those raises the number, and every one of them moves people from the middle row into the bottom row.
Nobody sets out to do this. It happens because the metric is on a dashboard with a target next to it, and the behaviours that satisfy it are easier than the behaviours that would genuinely help.
The two failures do not cost the same
Underneath the measurement problem is a pricing problem. A deflection business case implicitly treats every avoided agent contact as the same saving. It is not.
An escalation costs an agent's handling time. That figure is known, modest, and already in your operating model.
A confident wrong answer costs whatever the customer then does. They act on incorrect information about a fee, a deadline, an entitlement or a cancellation right. In an unregulated business that is a complaint and some goodwill. In financial services it can be a remediation exercise and a reportable event, and it will not be discovered on the day it happens.
So the bot that escalates slightly too often is making a cheap mistake, and the bot that answers slightly too often is making an expensive one. Containment treats them as opposites of equal weight, which is how a well-intentioned target ends up causing the harm it was meant to avoid.
Measure the thing that is hard to fake
Repeat contact rate is the most useful number available, and most organisations can already produce it. Of the conversations the bot contained, what share of those people got back in touch through any channel within the next few days?
It works because it is behavioural rather than declared. It needs no survey. And it is genuinely difficult to game: making escalation harder inside the bot does nothing to stop someone picking up the phone tomorrow. Anything that inflates containment dishonestly shows up here as a worse number, which is exactly the property you want in a counterweight.
Two things to pair with it:
Cost per resolved contact, not cost per conversation. The denominator change is the whole point. It stops a cheap non-answer counting as efficiency.
The refusal rate — how often the bot says it is not sure and routes onward. This should be monitored and should never be zero. A language model will produce an answer rather than decline unless it is designed not to, so a bot that never expresses uncertainty is not more capable; it is less honest. The same logic applies wherever a model acts rather than answers, which we set out in where an AI agent should stop and ask.
On satisfaction scores
Worth a short note because they are often offered as the answer to all this. Post-conversation satisfaction surveys are weak evidence on their own, because of who responds: contented people rarely bother, irritated people sometimes do, and the customers who abandoned the conversation are by definition not there to be asked.
They are useful as a supporting signal and for the free-text comments, which frequently tell you more than the score. They are not a substitute for watching what people actually did next.
What a good target looks like
- Never set containment alone as a target. If it is reported, report repeat contact rate beside it, always, on the same chart.
- Separate abandonment from resolution in the analytics, even approximately — a conversation ending mid-exchange with no confirmation is not the same as one ending after an answer.
- Price the two failure modes separately in the business case, especially in a regulated context.
- Make escalation easy and visible, and accept the lower containment number that follows.
- Watch the refusal rate for drift. A falling one usually means the bot is guessing more, not knowing more.
- Read the transcripts of contained conversations, a sample every week. It is unglamorous and it finds things no metric will.
A bot measured this way will report a less impressive headline than one measured on containment alone. It will also be the one customers do not complain about, which is the outcome the original business case was actually trying to buy.
Questions we get asked
What is chatbot containment rate?
The proportion of conversations that ended without reaching a human agent. It is the most commonly reported chatbot metric because it is trivial to calculate from data you already have. Its defect is that it cannot distinguish a customer whose problem was solved from one who gave up, since neither of them contacted an agent.
Why is containment a misleading measure?
Because the easiest way to raise it is to make escalation harder. A bot that hides the route to a human, or that answers confidently enough to end the conversation without solving anything, scores better than one that escalates early and honestly. The metric rewards the behaviour you least want, which is why it should never be a target on its own.
What should you measure instead?
Repeat contact rate is the single most useful signal: the share of people who come back through any channel within a few days of a contained conversation. It is hard to game, it needs no survey, and it approximates the question you actually care about — was the problem solved. Pair it with cost per resolved contact rather than cost per conversation.
Should a chatbot be able to say it does not know?
Yes, and the rate at which it does should be monitored and should never be zero. A language model will produce an answer rather than decline unless designed otherwise, so a bot that never says it is unsure is not more capable — it is less honest. Treat a falling refusal rate as a warning rather than as progress.
Is a wrong answer worse than escalating to a human?
Almost always, and the gap widens with regulation. An escalation costs an agent's time, which is a known and modest figure. A confident wrong answer about a fee, an entitlement or a deadline can cost a complaint, a remediation, and in a regulated sector a reportable event. Pricing those two failures identically is how containment targets end up causing harm.
How reliable is customer satisfaction after a bot conversation?
Weak on its own, because of who responds. Satisfied people rarely fill in surveys, frustrated ones sometimes do, and those who abandoned the conversation have already left. Use it as supporting evidence rather than a primary measure, and put more weight on behavioural signals like whether the person came back.
If you are building a chatbot business case, the cheapest improvement available is changing the denominator before anyone writes a prompt. That is the conversation we open with on chatbot projects, because the measurement decision shapes the product more than the model does.