Terrible — Bot fails and makes another mistake in a series — consumers may end the chat angrily.
Very Bad — Bot response fails to assist and doesn't attempt to resolve the problem.
Bad — Bot response misses the previous consumer response, yet might recover.
Satisfactory — Bot response is somewhat helpful but doesn't complete the task.
Good — Bot response is relevant and helpful, bringing the consumer closer to resolution.
The real Interaction Quality (IQ) annotation scale — a running score, rated turn by turn, no diagnosis attached.
01
A score tells you when. Not what.
Most task-oriented chatbots aren’t maintained by engineers — they’re maintained by bot-builders, customer service specialists who use drag-and-drop tools to build and adjust dialog flows. When a bot is doing badly, the standard signal is Interaction Quality (IQ), a running per-turn score on the 5-point scale to the right.
IQ is genuinely useful. It tells you a conversation is going badly, and roughly when. What it never tells you is why, or what to actually do about it. Without that, a bot-builder’s only option is manually reviewing transcripts, which is work that “typically takes days or weeks.”
Hi! I’m the airline’s virtual assistant. How can I help with your trip today?
IQ: Satisfactory
Step through the conversation. The score shows when it goes wrong.
An illustrative airline conversation, not from the paper’s data, annotated the paper’s way (Appendix A’s rules, Table 9’s five levels, Table 10’s categories); fixes quoted from Table 2. The same falling line has three different causes, and each needs a different fix.
02
So here’s what: Actionable Conversational Quality Indicators
With colleagues at LivePerson, I helped build a taxonomy of 9 real categories ACQIs that name the specific way a turn went wrong (or right), each tied to a concrete set of fixes a bot-builder can actually take. “Does Not Understand” means something different, and needs a different fix, than “Bad Transfer” or “Ignored Consumer Statement.” That’s the whole idea: not a better score, a better diagnosis.
Below is the real taxonomy: all 9 categories, their real definitions, and a few of each one’s real recommended actions, not paraphrased into something vaguer.
All 9 categories, real descriptions, and a few of each one’s real recommended actions — the full taxonomy has 28–31 possible actions across all ACQIs, depending on the dialog system. Click a category to follow it through every chart below.
Total
Does Not Understand
Bad Transfer
Input Rejected
Ignored Consumer Statement
Unable to Resolve
Restart
Provides Assistance
Ask for Information
Ask for Confirmation
Real proportions from the paper’s own annotated data (Figure 5) — most ACQIs skew negative, but not all, and not always.
03
Does it actually hold up?
Three expert annotators labeled ACQI and IQ together across 541 real conversations from a public transit-info bot (LEGOv2) and 520 real conversations from four different LivePerson production bots in retail, travel, tech support, food ordering. Their agreement on which ACQI applied to a given turn was a Cohen’s Kappa of .68, substantial agreement for a judgment call, not a coin flip.
And the categories aren’t simply “bad”, most skew negative, but not all, and not always. A concrete case: for most bots, a falling score is dominated by “Does Not Understand.” For the retail bot specifically, 54% of its score drops were “Provides Assistance” instead, the bot was actively over-helping, offering assistance nobody asked for. Same symptom, a completely different real cause, a completely different fix.
LEGOv2 (bus info, spoken) — top cause of score drops: Does Not Understand, 65.5%
Share of each bot’s IQ score drops, by ACQI (Table 5). Same symptom, different cause — pick a bot.
Asking for the 1st time moves the score by +0.34 on average (95%: +0.28 to +0.41). Still reliably helps.
“Ask for Confirmation” by repetition (Figure 6, values digitized from the paper’s own chart). One confirmation helps most; by the 7th, the benefit can’t be told apart from harm.
04
Can a model find these on its own?
Annotating every conversation by hand doesn’t scale. A text-only classifier (Sentence-BERT turn embeddings feeding a logistic regression, nothing exotic) predicts the correct ACQI at a 0.790 weighted-average F1 across all 9 categories, and the exact IQ score around 60% of the time. It’s also cheap to train: performance is already near its ceiling with only about 125 conversations, not thousands.
Per-ACQI F1 predicting from text alone (Table 7), every category weighted the same. The rare ones drag the average down to 0.574.
Real performance vs. training-set size (Figure 7) — all four metrics are near their ceiling by ~125 conversations.
Real numbers, Table 8 — average action-strategies per turn, with 95% confidence intervals (bootstrapped at the turn level).
05
The payoff
With no measurement at all, a bot-builder facing a bad turn has to weigh roughly 28.6 possible action strategies, on average, across the dialog systems studied, an exhaustive, unguided search. Knowing the IQ score alone narrows that to 9.73. Add ACQI on top of it and it drops to 5.40, an ~81% reduction from the unguided baseline.
That’s the real, hypothetical-but-grounded case for the whole approach: if IQ and ACQI can be predicted accurately, the amount of manual triage a bot-builder has to do on any given bad turn drops by roughly four fifths. Even when the predictions aren’t perfect, they still narrow the search. A bot-builder does no worse than the exhaustive default, and does much better whenever the prediction lands.
28.60
actions to weigh per turn
of 28.6 possible
No measurement: every turn is suspect, and every action is a candidate for every turn.
Averages per turn from the paper’s own simulation (Table 8, 95% bootstrap intervals); actions from Table 2.
06
Where it doesn’t (yet) hold up
The honest limitation: these models generalize well within LivePerson’s own bots, a model trained on three industries and tested on a fourth still reaches a macro F1 between 0.69 and 0.85. But they don’t transfer between LivePerson’s data and the public LEGOv2 dataset in either direction, macro F1 drops as low as 0.10–0.16. A model that understands “does not understand” in a retail chatbot doesn’t automatically understand it in a phone-based bus-information system, and figuring out why is real, unfinished future work, not something this paper claims to have solved.
Click an industry to hold it out; the model trains on the other three.
Model: LivePerson’s production ACQI model (Table 11)
every category counts once
every bot turn counts once
new domainsame company,
new industry1
Every run’s macro F1 (Appendix B, Tables 11 and 13). Within one company’s bots the model transfers; across companies and channels it doesn’t.
This work was done with colleagues at LivePerson: Michael Higgins, Dominic Widdows, Beth Ann Hockey, Akshay Hazare, Kristen Howell, Gwen Christian, Sujit Mathi, Chris Brew, Andrew Maurer, Matthew Dunn, and Joe Bradley. It was submitted and accepted to Natural Language Engineering (Cambridge University Press).
A coarse running score, fused with a specific diagnostic category, into one answer a bot-builder can actually act on, that’s the same type of problem as anywhere else on this site: combine a fast, general signal with a slower, specific one, and the fusion is worth more than either alone.