# Taming Repetition in Dialogue Generation

arXiv: https://arxiv.org/abs/2112.08657 · HTML: https://mlsanity.com/paper/2112.08657/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 28 of its 37 references that are in the map.
- First posted on arXiv 16 Dec 2021.

## Builds on (most cited first, 28 of 28)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [The Curious Case of Neural Text Degeneration](https://mlsanity.com/paper/1904.09751/) (2019)
- [fairseq: A Fast, Extensible Toolkit for Sequence Modeling](https://mlsanity.com/paper/1904.01038/) (2019)
- [Get To The Point: Summarization with Pointer-Generator Networks](https://mlsanity.com/paper/1704.04368/) (2017)
- [A Diversity-Promoting Objective Function for Neural Conversation Models](https://mlsanity.com/paper/1510.03055/) (2015)
- [DialoGPT: Large-Scale Generative Pre-training for Conversational Response Generation](https://mlsanity.com/paper/1911.00536/) (2019)
- [A Survey on Multi-Task Learning](https://mlsanity.com/paper/1707.08114/) (2017)
- [Scheduled Sampling for Sequence Prediction with Recurrent Neural Networks](https://mlsanity.com/paper/1506.03099/) (2015)
- [CTRL: A Conditional Transformer Language Model for Controllable Generation](https://mlsanity.com/paper/1909.05858/) (2019)
- [Sequence Level Training with Recurrent Neural Networks](https://mlsanity.com/paper/1511.06732/) (2015)
- [Neural Text Generation with Unlikelihood Training](https://mlsanity.com/paper/1908.04319/) (2019)
- [Towards a Human-like Open-Domain Chatbot](https://mlsanity.com/paper/2001.09977/) (2020)
- [Deep Reinforcement Learning for Dialogue Generation](https://mlsanity.com/paper/1606.01541/) (2016)
- [Beyond Goldfish Memory: Long-Term Open-Domain Conversation](https://mlsanity.com/paper/2107.07567/) (2021)
- [OpenNMT: Open-source Toolkit for Neural Machine Translation](https://mlsanity.com/paper/1709.03815/) (2017)
- [A Neural Conversational Model](https://mlsanity.com/paper/1506.05869/) (2015)
- [TransferTransfo: A Transfer Learning Approach for Neural Network Based Conversational Agents](https://mlsanity.com/paper/1901.08149/) (2019)
- [Bottom-Up Abstractive Summarization](https://mlsanity.com/paper/1808.10792/) (2018)
- [What makes a good conversation? How controllable attributes affect human judgments](https://mlsanity.com/paper/1902.08654/) (2019)
- [Unifying Human and Statistical Evaluation for Natural Language Generation](https://mlsanity.com/paper/1904.02792/) (2019)
- [Modeling Coverage for Neural Machine Translation](https://mlsanity.com/paper/1601.04811/) (2016)
- [Do Massively Pretrained Language Models Make Better Storytellers?](https://mlsanity.com/paper/1909.10705/) (2019)
- [Diversifying Dialog Generation via Adaptive Label Smoothing](https://mlsanity.com/paper/2105.14556/) (2021)
- [Cutting-off Redundant Repeating Generations for Neural Abstractive Summarization](https://mlsanity.com/paper/1701.00138/) (2016)

## Built on by (most cited first, 5 of 5)

- [Leveraging Large Language Models for Automated Dialogue Analysis](https://mlsanity.com/paper/2309.06490/) (2023)
- [Repetitions are not all alike: distinct mechanisms sustain repetition in language models](https://mlsanity.com/paper/2504.01100/) (2025)
- [Construction Repetition Reduces Information Rate in Dialogue](https://mlsanity.com/paper/2210.08321/) (2022)
- [Attribution and Alignment: Effects of Local Context Repetition on Utterance Production and Comprehension in Dialogue](https://mlsanity.com/paper/2311.13061/) (2023)
- [Mitigating the Learning Bias towards Repetition by Self-Contrastive Training for Open-Ended Generation](https://mlsanity.com/paper/2307.01542/) (2023)

## Abstract (arXiv)

The wave of pre-training language models has been continuously improving the quality of the machine-generated conversations, however, some of the generated responses still suffer from excessive repetition, sometimes repeating words from utterance, sometimes repeating words within self-generated responses, or both. Inappropriate repetition of words can significantly degrade the quality of the generated texts. Penalized sampling is one popular solution, reducing the sampling probability of existing words during inference, however, it is highly vulnerable to the inappropriate setting of the static weight. Setting it too high can yield strange and unrealistic sentences while setting it too low makes the task of suppressing repetition trivial. To remedy the shortcomings of the above methods, we design a context-aware classifier to explicitly decide when to allow repetition and when to employ penalized sampling. Such a classifier can be easily integrated with existing decoding methods, reducing repetitions where appropriate while preserving the diversity of the text. Experimental results demonstrate that our method can generate higher quality and more authentic dialogues.