# Do Neural Dialog Systems Use the Conversation History Effectively? An Empirical Study

arXiv: https://arxiv.org/abs/1906.01603 · HTML: https://mlsanity.com/paper/1906.01603/

## Key facts

- Cited by 47 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 4 Jun 2019.

## Built on by (most cited first, 30 of 47)

- [Lost in the Middle: How Language Models Use Long Contexts](https://mlsanity.com/paper/2307.03172/) (2023)
- [Dynabench: Rethinking Benchmarking in NLP](https://mlsanity.com/paper/2104.14337/) (2021)
- [Recent Advances in Deep Learning Based Dialogue Systems: A Systematic Survey](https://mlsanity.com/paper/2105.04387/) (2021)
- [Anticipating Safety Issues in E2E Conversational AI: Framework and Tooling](https://mlsanity.com/paper/2107.03451/) (2021)
- [What Context Features Can Transformer Language Models Use?](https://mlsanity.com/paper/2106.08367/) (2021)
- [Conversations Are Not Flat: Modeling the Dynamic Information Flow across Dialogue Utterances](https://mlsanity.com/paper/2106.02227/) (2021)
- [Coherence boosting: When your pretrained language model is not paying enough attention](https://mlsanity.com/paper/2110.08294/) (2021)
- [ConvXAI: Delivering Heterogeneous AI Explanations via Conversations to Support Human-AI Scientific Writing](https://mlsanity.com/paper/2305.09770/) (2023)
- [A Model-Agnostic Data Manipulation Method for Persona-based Dialogue Generation](https://mlsanity.com/paper/2204.09867/) (2022)
- [Investigating Robustness of Dialog Models to Popular Figurative Language Constructs](https://mlsanity.com/paper/2110.00687/) (2021)
- [VCSUM: A Versatile Chinese Meeting Summarization Dataset](https://mlsanity.com/paper/2305.05280/) (2023)
- [A Mixture-of-Expert Approach to RL-based Dialogue Management](https://mlsanity.com/paper/2206.00059/) (2022)
- [Probing Commonsense Explanation in Dialogue Response Generation](https://mlsanity.com/paper/2104.09574/) (2021)
- [Learning from Perturbations: Diverse and Informative Dialogue Generation with Inverse Adversarial Training](https://mlsanity.com/paper/2105.15171/) (2021)
- [Local Structure Matters Most: Perturbation Study in NLU](https://mlsanity.com/paper/2107.13955/) (2021)
- [Code-switched inspired losses for generic spoken dialog representations](https://mlsanity.com/paper/2108.12465/) (2021)
- [Shaking Syntactic Trees on the Sesame Street: Multilingual Probing with Controllable Perturbations](https://mlsanity.com/paper/2109.14017/) (2021)
- [Less is More: Mitigate Spurious Correlations for Open-Domain Dialogue Response Generation Models by Causal Discovery](https://mlsanity.com/paper/2303.01962/) (2023)
- [Integrating Dialog History into End-to-End Spoken Language Understanding Systems](https://mlsanity.com/paper/2108.08405/) (2021)
- [Know Thy Strengths: Comprehensive Dialogue State Tracking Diagnostics](https://mlsanity.com/paper/2112.08321/) (2021)
- [Back to the Future: Bidirectional Information Decoupling Network for Multi-turn Dialogue Modeling](https://mlsanity.com/paper/2204.08152/) (2022)
- [Don't Take This Out of Context! On the Need for Contextual Models and Evaluations for Stylistic Rewriting](https://mlsanity.com/paper/2305.14755/) (2023)
- [Explaining the Road Not Taken](https://mlsanity.com/paper/2103.14973/) (2021)
- [Sometimes We Want Translationese](https://mlsanity.com/paper/2104.07623/) (2021)
- [Joint Modelling of Spoken Language Understanding Tasks with Integrated Dialog History](https://mlsanity.com/paper/2305.00926/) (2023)
- [Analyzing Norm Violations in Live-Stream Chat](https://mlsanity.com/paper/2305.10731/) (2023)
- [Few-Shot Dialogue Summarization via Skeleton-Assisted Prompt Transfer in Prompt Tuning](https://mlsanity.com/paper/2305.12077/) (2023)
- [Revisiting Topic-Guided Language Models](https://mlsanity.com/paper/2312.02331/) (2023)
- [Enhancing Code Generation via Bidirectional Comment-Level Mutual Grounding](https://mlsanity.com/paper/2505.07768/) (2025)
- [Does Dialog Length matter for Next Response Selection task? An Empirical Study](https://mlsanity.com/paper/2101.09647/) (2021)

## Abstract (arXiv)

Neural generative models have been become increasingly popular when building conversational agents. They offer flexibility, can be easily adapted to new domains, and require minimal domain engineering. A common criticism of these systems is that they seldom understand or use the available dialog history effectively. In this paper, we take an empirical approach to understanding how these models use the available dialog history by studying the sensitivity of the models to artificially introduced unnatural changes or perturbations to their context at test time. We experiment with 10 different types of perturbations on 4 multi-turn dialog datasets and find that commonly used neural dialog architectures like recurrent and transformer-based seq2seq models are rarely sensitive to most perturbations such as missing or reordering utterances, shuffling words, etc. Also, by open-sourcing our code, we believe that it will serve as a useful diagnostic tool for evaluating dialog systems in the future.