# Evaluating Open-Domain Dialogues in Latent Space with Next Sentence Prediction and Mutual Information

arXiv: https://arxiv.org/abs/2305.16967 · HTML: https://mlsanity.com/paper/2305.16967/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 25 of its 39 references that are in the map.
- First posted on arXiv 26 May 2023.

## Builds on (most cited first, 25 of 25)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [BERTScore: Evaluating Text Generation with BERT](https://mlsanity.com/paper/1904.09675/) (2019)
- [Sequence to Sequence Learning with Neural Networks](https://mlsanity.com/paper/1409.3215/) (2014)
- [BLEURT: Learning Robust Metrics for Text Generation](https://mlsanity.com/paper/2004.04696/) (2020)
- [A Diversity-Promoting Objective Function for Neural Conversation Models](https://mlsanity.com/paper/1510.03055/) (2015)
- [Personalizing Dialogue Agents: I have a dog, do you have pets too?](https://mlsanity.com/paper/1801.07243/) (2018)
- [DailyDialog: A Manually Labelled Multi-turn Dialogue Dataset](https://mlsanity.com/paper/1710.03957/) (2017)
- [Formal Limitations on the Measurement of Mutual Information](https://mlsanity.com/paper/1811.04251/) (2018)
- [A Hierarchical Latent Variable Encoder-Decoder Model for Generating Dialogues](https://mlsanity.com/paper/1605.06069/) (2016)
- [Challenges in Building Intelligent Open-domain Dialog Systems](https://mlsanity.com/paper/1905.05709/) (2019)
- [Optimus: Organizing Sentences via Pre-trained Modeling of a Latent Space](https://mlsanity.com/paper/2004.04092/) (2020)
- [RUBER: An Unsupervised Method for Automatic Evaluation of Open-Domain Dialog Systems](https://mlsanity.com/paper/1701.03079/) (2017)
- [Towards Universal Paraphrastic Sentence Embeddings](https://mlsanity.com/paper/1511.08198/) (2015)
- [Improving Dialog Evaluation with a Multi-reference Adversarial Dataset and Large Scale Pretraining](https://mlsanity.com/paper/2009.11321/) (2020)
- [Better Automatic Evaluation of Open-Domain Dialogue Systems with Contextualized Embeddings](https://mlsanity.com/paper/1904.10635/) (2019)
- [Learning an Unreferenced Metric for Online Dialogue Evaluation](https://mlsanity.com/paper/2005.00583/) (2020)
- [Content preserving text generation with attribute controls](https://mlsanity.com/paper/1811.01135/) (2018)
- [DialogWAE: Multimodal Response Generation with Conditional Wasserstein Auto-Encoder](https://mlsanity.com/paper/1805.12352/) (2018)
- [Designing Precise and Robust Dialogue Response Evaluators](https://mlsanity.com/paper/2004.04908/) (2020)
- [A Surprisingly Effective Fix for Deep Latent Variable Modeling of Text](https://mlsanity.com/paper/1909.00868/) (2019)
- [MDD-Eval: Self-Training on Augmented Data for Multi-Domain Dialogue Evaluation](https://mlsanity.com/paper/2112.07194/) (2021)
- [Fast and Scalable Dialogue State Tracking with Explicit Modular Decomposition](https://mlsanity.com/paper/2004.10663/) (2020)
- [Improving Variational Autoencoder for Text Modelling with Timestep-Wise Regularisation](https://mlsanity.com/paper/2011.01136/) (2020)
- [Terminology-aware Medical Dialogue Generation](https://mlsanity.com/paper/2210.15551/) (2022)
- [Improving Variational Autoencoders with Density Gap-based Regularization](https://mlsanity.com/paper/2211.00321/) (2022)

## Built on by (most cited first, 6 of 6)

- [Improving Medical Dialogue Generation with Abstract Meaning Representations](https://mlsanity.com/paper/2309.10608/) (2023)
- [Beyond One-Size-Fits-All: Inversion Learning for Highly Effective NLG Evaluation Prompts](https://mlsanity.com/paper/2504.21117/) (2025)
- [ComperDial: Commonsense Persona-grounded Dialogue Dataset and Benchmark](https://mlsanity.com/paper/2406.11228/) (2024)
- [DRE: An Effective Dual-Refined Method for Integrating Small and Large Language Models in Open-Domain Dialogue Evaluation](https://mlsanity.com/paper/2506.04516/) (2025)
- [Why Text Prevails: Vision May Undermine Multimodal Medical Decision Making](https://mlsanity.com/paper/2512.13747/) (2025)
- [Natural Language Generation](https://mlsanity.com/paper/2503.16728/) (2025)

## Abstract (arXiv)

The long-standing one-to-many issue of the open-domain dialogues poses significant challenges for automatic evaluation methods, i.e., there may be multiple suitable responses which differ in semantics for a given conversational context. To tackle this challenge, we propose a novel learning-based automatic evaluation metric (CMN), which can robustly evaluate open-domain dialogues by augmenting Conditional Variational Autoencoders (CVAEs) with a Next Sentence Prediction (NSP) objective and employing Mutual Information (MI) to model the semantic similarity of text in the latent space. Experimental results on two open-domain dialogue datasets demonstrate the superiority of our method compared with a wide range of baselines, especially in handling responses which are distant to the golden reference responses in semantics.