# Interactive Evaluation of Dialog Track at DSTC9

arXiv: https://arxiv.org/abs/2207.14403 · HTML: https://mlsanity.com/paper/2207.14403/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- Builds on 19 of its 26 references that are in the map.
- First posted on arXiv 28 Jul 2022.

## Builds on (most cited first, 19 of 19)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [BERTScore: Evaluating Text Generation with BERT](https://mlsanity.com/paper/1904.09675/) (2019)
- [DialoGPT: Large-Scale Generative Pre-training for Conversational Response Generation](https://mlsanity.com/paper/1911.00536/) (2019)
- [Recipes for building an open-domain chatbot](https://mlsanity.com/paper/2004.13637/) (2020)
- [Towards a Human-like Open-Domain Chatbot](https://mlsanity.com/paper/2001.09977/) (2020)
- [How NOT To Evaluate Your Dialogue System: An Empirical Study of Unsupervised Evaluation Metrics for Dialogue Response Generation](https://mlsanity.com/paper/1603.08023/) (2016)
- [A Neural Conversational Model](https://mlsanity.com/paper/1506.05869/) (2015)
- [USR: An Unsupervised and Reference Free Evaluation Metric for Dialog Generation](https://mlsanity.com/paper/2005.00456/) (2020)
- [The Second Conversational Intelligence Challenge (ConvAI2)](https://mlsanity.com/paper/1902.00098/) (2019)
- [Unsupervised Evaluation of Interactive Dialog with DialoGPT](https://mlsanity.com/paper/2006.12719/) (2020)
- [Towards an Automatic Turing Test: Learning to Evaluate Dialogue Responses](https://mlsanity.com/paper/1708.07149/) (2017)
- [PLATO-2: Towards Building an Open-Domain Chatbot via Curriculum Learning](https://mlsanity.com/paper/2006.16779/) (2020)
- [Conversational AI: The Science Behind the Alexa Prize](https://mlsanity.com/paper/1801.03604/) (2018)
- [Learning an Unreferenced Metric for Online Dialogue Evaluation](https://mlsanity.com/paper/2005.00583/) (2020)
- [Overview of the Ninth Dialog System Technology Challenge: DSTC9](https://mlsanity.com/paper/2011.06486/) (2020)
- [Investigating Evaluation of Open-Domain Dialogue Systems With Human Generated Multiple References](https://mlsanity.com/paper/1907.10568/) (2019)
- [Advancing the State of the Art in Open Domain Dialog Systems through the Alexa Prize](https://mlsanity.com/paper/1812.10757/) (2018)
- [DialPort: Connecting the Spoken Dialog Research Community to Real User Data](https://mlsanity.com/paper/1606.02562/) (2016)
- [Beyond Turing: Intelligent Agents Centered on the User](https://mlsanity.com/paper/1901.06613/) (2019)

## Built on by (most cited first, 10 of 10)

- [SOTOPIA: Interactive Evaluation for Social Intelligence in Language Agents](https://mlsanity.com/paper/2310.11667/) (2023)
- [LLM-Eval: Unified Multi-Dimensional Automatic Evaluation for Open-Domain Conversations with Large Language Models](https://mlsanity.com/paper/2305.13711/) (2023)
- [SimulatorArena: Are User Simulators Reliable Proxies for Multi-Turn Evaluation of AI Assistants?](https://mlsanity.com/paper/2510.05444/) (2025)
- [A Comprehensive Analysis of the Effectiveness of Large Language Models as Automatic Dialogue Evaluators](https://mlsanity.com/paper/2312.15407/) (2023)
- [xDial-Eval: A Multilingual Open-Domain Dialogue Evaluation Benchmark](https://mlsanity.com/paper/2310.08958/) (2023)
- [Interaction Matters: An Evaluation Framework for Interactive Dialogue Assessment on English Second Language Conversations](https://mlsanity.com/paper/2407.06479/) (2024)
- [ComperDial: Commonsense Persona-grounded Dialogue Dataset and Benchmark](https://mlsanity.com/paper/2406.11228/) (2024)
- [Evaluating Agent Interactions Through Episodic Knowledge Graphs](https://mlsanity.com/paper/2209.11746/) (2022)
- [Overview of Dialog System Evaluation Track: Dimensionality, Language, Culture and Safety at DSTC 12](https://mlsanity.com/paper/2509.13569/) (2025)
- [The DialPort tools](https://mlsanity.com/paper/2208.10918/) (2022)

## Abstract (arXiv)

The ultimate goal of dialog research is to develop systems that can be effectively used in interactive settings by real users. To this end, we introduced the Interactive Evaluation of Dialog Track at the 9th Dialog System Technology Challenge. This track consisted of two sub-tasks. The first sub-task involved building knowledge-grounded response generation models. The second sub-task aimed to extend dialog models beyond static datasets by assessing them in an interactive setting with real users. Our track challenges participants to develop strong response generation models and explore strategies that extend them to back-and-forth interactions with real users. The progression from static corpora to interactive evaluation introduces unique challenges and facilitates a more thorough assessment of open-domain dialog systems. This paper provides an overview of the track, including the methodology and results. Furthermore, it provides insights into how to best evaluate open-domain dialog models