# Coherent Dialogue with Attention-based Language Models

arXiv: https://arxiv.org/abs/1611.06997 · HTML: https://mlsanity.com/paper/1611.06997/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 21 Nov 2016.

## Built on by (most cited first, 7 of 7)

- [CHAI: A CHatbot AI for Task-Oriented Dialogue with Offline Reinforcement Learning](https://mlsanity.com/paper/2204.08426/) (2022)
- [Context-Aware Language Modeling for Goal-Oriented Dialogue Systems](https://mlsanity.com/paper/2204.10198/) (2022)
- [Why is constrained neural language generation particularly challenging?](https://mlsanity.com/paper/2206.05395/) (2022)
- [Contextual Biasing of Language Models for Speech Recognition in Goal-Oriented Conversational Agents](https://mlsanity.com/paper/2103.10325/) (2021)
- [Multi-turn Dialogue Comprehension from a Topic-aware Perspective](https://mlsanity.com/paper/2309.09666/) (2023)
- [PICK: Polished & Informed Candidate Scoring for Knowledge-Grounded Dialogue Systems](https://mlsanity.com/paper/2309.10413/) (2023)
- [PersonaDrift: A Benchmark for Temporal Anomaly Detection in Language-Based Dementia Monitoring](https://mlsanity.com/paper/2511.16445/) (2025)

## Abstract (arXiv)

We model coherent conversation continuation via RNN-based dialogue models equipped with a dynamic attention mechanism. Our attention-RNN language model dynamically increases the scope of attention on the history as the conversation continues, as opposed to standard attention (or alignment) models with a fixed input scope in a sequence-to-sequence model. This allows each generated word to be associated with the most relevant words in its corresponding conversation history. We evaluate the model on two popular dialogue datasets, the open-domain MovieTriples dataset and the closed-domain Ubuntu Troubleshoot dataset, and achieve significant improvements over the state-of-the-art and baselines on several metrics, including complementary diversity-based metrics, human evaluation, and qualitative visualizations. We also show that a vanilla RNN with dynamic attention outperforms more complex memory models (e.g., LSTM and GRU) by allowing for flexible, long-distance memory. We promote further coherence via topic modeling-based reranking.