# Topic-based Evaluation for Conversational Bots

arXiv: https://arxiv.org/abs/1801.03622 · HTML: https://mlsanity.com/paper/1801.03622/

## Key facts

- Cited by 16 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 11 Jan 2018.

## Built on by (most cited first, 16 of 16)

- [A Comprehensive Assessment of Dialog Evaluation Metrics](https://mlsanity.com/paper/2106.03706/) (2021)
- [Compression, Transduction, and Creation: A Unified Framework for Evaluating Natural Language Generation](https://mlsanity.com/paper/2109.06379/) (2021)
- [Leveraging Implicit Feedback from Deployment Data in Dialogue](https://mlsanity.com/paper/2307.14117/) (2023)
- [HERALD: An Annotation Efficient Method to Detect User Disengagement in Social Conversations](https://mlsanity.com/paper/2106.00162/) (2021)
- [State-of-the-art in Open-domain Conversational AI: A Survey](https://mlsanity.com/paper/2205.00965/) (2022)
- [Dr.Copilot: A Multi-Agent Prompt Optimized Assistant for Improving Patient-Doctor Communication in Romanian](https://mlsanity.com/paper/2507.11299/) (2025)
- [SWAN: A Generic Framework for Auditing Textual Conversational Systems](https://mlsanity.com/paper/2305.08290/) (2023)
- [The Interplay of Task Success and Dialogue Quality: An in-depth Evaluation in Task-Oriented Visual Dialogues](https://mlsanity.com/paper/2103.11151/) (2021)
- [How to Evaluate Your Dialogue Models: A Review of Approaches](https://mlsanity.com/paper/2108.01369/) (2021)
- [KoDialogBench: Evaluating Conversational Understanding of Language Models with Korean Dialogue Benchmark](https://mlsanity.com/paper/2402.17377/) (2024)
- [FCC: Fusing Conversation History and Candidate Provenance for Contextual Response Ranking in Dialogue Systems](https://mlsanity.com/paper/2304.00180/) (2023)
- [Social Commonsense-Guided Search Query Generation for Open-Domain Knowledge-Powered Conversations](https://mlsanity.com/paper/2310.14340/) (2023)
- [HammerBench: Fine-Grained Function-Calling Evaluation in Real Mobile Device Scenarios](https://mlsanity.com/paper/2412.16516/) (2024)
- [Open-domain Topic Identification of Out-of-domain Utterances using Wikipedia](https://mlsanity.com/paper/2101.11134/) (2021)
- [QoNext: Towards Next-generation QoE for Foundation Models](https://mlsanity.com/paper/2509.21889/) (2025)
- [PMIScore: An Unsupervised Approach to Quantify Dialogue Engagement](https://mlsanity.com/paper/2603.13796/) (2026)

## Abstract (arXiv)

Dialog evaluation is a challenging problem, especially for non task-oriented dialogs where conversational success is not well-defined. We propose to evaluate dialog quality using topic-based metrics that describe the ability of a conversational bot to sustain coherent and engaging conversations on a topic, and the diversity of topics that a bot can handle. To detect conversation topics per utterance, we adopt Deep Average Networks (DAN) and train a topic classifier on a variety of question and query data categorized into multiple topics. We propose a novel extension to DAN by adding a topic-word attention table that allows the system to jointly capture topic keywords in an utterance and perform topic classification. We compare our proposed topic based metrics with the ratings provided by users and show that our metrics both correlate with and complement human judgment. Our analysis is performed on tens of thousands of real human-bot dialogs from the Alexa Prize competition and highlights user expectations for conversational bots.