# Improving Neural Conversational Models with Entropy-Based Data Filtering

arXiv: https://arxiv.org/abs/1905.05471 · HTML: https://mlsanity.com/paper/1905.05471/

## Key facts

- Cited by 30 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 14 May 2019.

## Built on by (most cited first, 30 of 30)

- [Beyond Goldfish Memory: Long-Term Open-Domain Conversation](https://mlsanity.com/paper/2107.07567/) (2021)
- [Recent Advances in Deep Learning Based Dialogue Systems: A Systematic Survey](https://mlsanity.com/paper/2105.04387/) (2021)
- [Bias in Large Language Models: Origin, Evaluation, and Mitigation](https://mlsanity.com/paper/2411.10915/) (2024)
- [Weakly Supervised Data Augmentation Through Prompting for Dialogue Understanding](https://mlsanity.com/paper/2210.14169/) (2022)
- [A Model-Agnostic Data Manipulation Method for Persona-based Dialogue Generation](https://mlsanity.com/paper/2204.09867/) (2022)
- [Generating Relevant and Coherent Dialogue Responses using Self-separated Conditional Variational AutoEncoders](https://mlsanity.com/paper/2106.03410/) (2021)
- [Measuring and Improving Semantic Diversity of Dialogue Generation](https://mlsanity.com/paper/2210.05725/) (2022)
- [Identifying Untrustworthy Samples: Data Filtering for Open-domain Dialogues with Bayesian Optimization](https://mlsanity.com/paper/2109.06471/) (2021)
- [Learning from Perturbations: Diverse and Informative Dialogue Generation with Inverse Adversarial Training](https://mlsanity.com/paper/2105.15171/) (2021)
- [Less is More: Mitigate Spurious Correlations for Open-Domain Dialogue Response Generation Models by Causal Discovery](https://mlsanity.com/paper/2303.01962/) (2023)
- [Dynamic Stochastic Decoding Strategy for Open-Domain Dialogue Generation](https://mlsanity.com/paper/2406.07850/) (2024)
- [Towards Efficiently Diversifying Dialogue Generation via Embedding Augmentation](https://mlsanity.com/paper/2103.01534/) (2021)
- [CARE: Causality Reasoning for Empathetic Responses by Conditional Graph Generation](https://mlsanity.com/paper/2211.00255/) (2022)
- [Improved Instruction Ordering in Recipe-Grounded Conversation](https://mlsanity.com/paper/2305.17280/) (2023)
- [Diversifying Neural Dialogue Generation via Negative Distillation](https://mlsanity.com/paper/2205.02795/) (2022)
- [Deal, or no deal (or who knows)? Forecasting Uncertainty in Conversations using Large Language Models](https://mlsanity.com/paper/2402.03284/) (2024)
- [Multi-View Feature Representation for Dialogue Generation with Bidirectional Distillation](https://mlsanity.com/paper/2102.10780/) (2021)
- [Towards Diverse, Relevant and Coherent Open-Domain Dialogue Generation via Hybrid Latent Variables](https://mlsanity.com/paper/2212.01145/) (2022)
- [Learning towards Selective Data Augmentation for Dialogue Generation](https://mlsanity.com/paper/2303.09719/) (2023)
- [Stop Filtering: Multi-View Attribute-Enhanced Dialogue Learning](https://mlsanity.com/paper/2205.11206/) (2022)
- [Exposing Long-Tail Safety Failures in Large Language Models through Efficient Diverse Response Sampling](https://mlsanity.com/paper/2603.14355/) (2026)
- [Modeling Complex Dialogue Mappings via Sentence Semantic Segmentation Guided Conditional Variational Auto-Encoder](https://mlsanity.com/paper/2212.00231/) (2022)
- [Heterogeneous-Branch Collaborative Learning for Dialogue Generation](https://mlsanity.com/paper/2303.11621/) (2023)
- [Solving LLM Repetition Problem in Production: A Comprehensive Study of Multiple Solutions](https://mlsanity.com/paper/2512.04419/) (2025)
- [A Neural Conversation Generation Model via Equivalent Shared Memory Investigation](https://mlsanity.com/paper/2108.09164/) (2021)
- [Relevance in Dialogue: Is Less More? An Empirical Comparison of Existing Metrics, and a Novel Simple Metric](https://mlsanity.com/paper/2206.01823/) (2022)
- [Identifying Semantically Difficult Samples to Improve Text Classification](https://mlsanity.com/paper/2302.06155/) (2023)
- [Dior-CVAE: Pre-trained Language Models and Diffusion Priors for Variational Dialog Generation](https://mlsanity.com/paper/2305.15025/) (2023)
- [Learning to Diversify Neural Text Generation via Degenerative Model](https://mlsanity.com/paper/2309.12619/) (2023)
- [ConvoCache: Smart Re-Use of Chatbot Responses](https://mlsanity.com/paper/2406.18133/) (2024)

## Abstract (arXiv)

Current neural network-based conversational models lack diversity and generate boring responses to open-ended utterances. Priors such as persona, emotion, or topic provide additional information to dialog models to aid response generation, but annotating a dataset with priors is expensive and such annotations are rarely available. While previous methods for improving the quality of open-domain response generation focused on either the underlying model or the training objective, we present a method of filtering dialog datasets by removing generic utterances from training data using a simple entropy-based approach that does not require human supervision. We conduct extensive experiments with different variations of our method, and compare dialog models across 17 evaluation metrics to show that training on datasets filtered this way results in better conversational quality as chatbots learn to output more diverse responses.