What should we automate first?
Start with frequent, well-defined cases and an available source of truth: explaining a procedure, finding a status, collecting missing details or routing correctly. The most expensive complaint may be a poor first pilot because it needs individual judgement.
Take a sample of real requests and group them by reason for contact. Record each group’s frequency, handling time, system-access needs and consequences of error. Also track repeat contact. Repeated simple questions may indicate a product issue or missing website information.
The automation need not be visible to customers. Preparing material for an operator may provide more value than a standalone bot reply. Remove the cause of delay rather than merely adding another communication window.
What does research say about AI’s value in support?
An NBER study examined assistant adoption among 5,179 support workers. It reports an average increase in productivity, measured as cases resolved per hour, of 14 % and an increase for less experienced or lower-performing workers of 34 %. More experienced staff saw substantially smaller gains. This concerns AI assisting people, not proof that an autonomous bot resolves the same share of cases. Brynjolfsson, Li and Raymond: Generative AI at Work, 2023 revision.
“With large heterogeneity in effects across workers.” — Erik Brynjolfsson, Danielle Li and Lindsey Raymond highlight differences in effects between workers.
Our practical recommendation is to evaluate groups of cases and staff separately. An average saving may conceal a system that helps beginners but slows complaint specialists with unsuitable drafts they must check.
How do answers, staff assistance and actual resolution differ?
Each category has a different outcome and review cost. Do not combine them into one “automated support” percentage.
| Use case | What the system does | What remains |
|---|---|---|
| Classification | Identifies the topic and routes the case | Specialist resolution |
| Reply draft | Prepares text and supporting material | Operator review and sending |
| Independent answer | Explains supported information | Checking whether it was sufficient |
| Service agent | Performs an authorised system change | Outcome confirmation and records |
A similar distinction appears in a July 2026 CustomerSuccess discussion. Participants note that summarising or routing is not the same as resolution. These are individual opinions, not representative statistics.
Add separate labels to your case records. Then decide whether to expand autonomy, improve the knowledge base or adjust handovers between teams.
Why do handovers to humans frustrate customers?
Because customers often have to start over. The bot has their order number and problem description, but the operator cannot see them. Or the system closes the case while the customer repeatedly says the answer did not help.
In the Intercom Community in June 2026 users discuss repeated negative customer responses when asked whether Fin answered their question. This suggests a useful test: disagreement must change the next action. Rewording the same answer is not automatically progress towards resolution.
A handed-over case should contain the original request, verified details, source used, actions taken and remaining decisions. Give the customer realistic information about what follows. If human staff are absent, the agent must not pretend to transfer immediately.
How should we prepare data for good answers?
Start with the rules staff actually use. Prices, delivery terms and instructions need an owner and a current version. Historical conversations may show a useful approach or an exception that expired long ago.
Separate public information from customer-specific data. A general complaints procedure can be explained to anyone. A specific complaint’s status requires appropriate identification. Knowing an order number alone may not adequately verify someone for access to further personal data.
Every answer should have traceable support. For staff drafts, the source can appear beside the text. For customers, link to a relevant guide without overwhelming them with internal document names.
How should we test quality before launch?
Prepare real, anonymised cases, including difficult ones. Include unclear requests, wrong order numbers, repeat complaints and requests outside the rules. Testing only well-phrased questions is insufficient.
The newer 2026 Anthropic methodology for evaluating agents emphasises evaluating both the trajectory and final state. In support, that means checking the correct customer, action and follow-up record, not just the wording.
Repeat relevant tests after changing knowledge or permissions. A new returns policy can invalidate a previously correct answer even when the model stays the same.
Which metrics reveal the real value?
Track resolution without repeat contact, action correctness, satisfaction and cost per completed case. Deduct corrections and reviews from time saved. Watch the slowest cases as well as averages, since they often cause escalations.
Illustrative example: of 1,000 monthly questions, 400 are suitable for independent handling. If the agent resolves 300 correctly, that is 30% of all questions, not 75% automation of the entire support operation. The other 100 eligible cases still need intervention. Reporting both figures prevents a marketing percentage from hiding the remaining workload.
End the pilot with a decision for a specific case group: expand, adjust or retain human handling. Only then add another channel or permissions to the AI customer support agent.
Frequently asked questions
Should we deploy a chatbot first?
Only if website chat is the main source of recurring questions. If most work arrives by email, classification and reply drafts in the existing helpdesk may deliver more value.
How do we prevent customers getting stuck with a bot?
Define handover rules based on topic, failed attempts and customer preference. A handover must create a specific case and preserve context; merely telling the customer to contact support is not enough.
Can support operate independently overnight?
Yes, for clearly defined cases with the necessary data and permissions. For other cases, the agent collects information and explains the real next step without inventing human availability.
Research and solution design: Tanduva with AI assistance. External case studies are identified; illustrative examples are not measured results from our clients. Editorial methodology and corrections.