# Don't Say That! Making Inconsistent Dialogue Unlikely with Unlikelihood Training

arXiv: https://arxiv.org/abs/1911.03860 · HTML: https://mlsanity.com/paper/1911.03860/

## Key facts

- Cited by 110 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 10 Nov 2019.

## Built on by (most cited first, 30 of 110)

- [Survey of Hallucination in Natural Language Generation](https://mlsanity.com/paper/2202.03629/) (2022)
- [Red Teaming Language Models with Language Models](https://mlsanity.com/paper/2202.03286/) (2022)
- [ORPO: Monolithic Preference Optimization without Reference Model](https://mlsanity.com/paper/2403.07691/) (2024)
- [Memory-Based Model Editing at Scale](https://mlsanity.com/paper/2206.06520/) (2022)
- [Quark: Controllable Text Generation with Reinforced Unlearning](https://mlsanity.com/paper/2205.13636/) (2022)
- [The Surprising Effectiveness of Negative Reinforcement in LLM Reasoning](https://mlsanity.com/paper/2506.01347/) (2025)
- [Chain of Hindsight Aligns Language Models with Feedback](https://mlsanity.com/paper/2302.02676/) (2023)
- [Recent Advances in Deep Learning Based Dialogue Systems: A Systematic Survey](https://mlsanity.com/paper/2105.04387/) (2021)
- [How much do language models copy from their training data? Evaluating linguistic novelty in text generation using RAVEN](https://mlsanity.com/paper/2111.09509/) (2021)
- [CLIFF: Contrastive Learning for Improving Faithfulness and Factuality in Abstractive Summarization](https://mlsanity.com/paper/2109.09209/) (2021)
- [Entity-level Factual Consistency of Abstractive Text Summarization](https://mlsanity.com/paper/2102.09130/) (2021)
- [Locally Typical Sampling](https://mlsanity.com/paper/2202.00666/) (2022)
- [Some things are more CRINGE than others: Iterative Preference Optimization with the Pairwise Cringe Loss](https://mlsanity.com/paper/2312.16682/) (2023)
- [BoB: BERT Over BERT for Training Persona-based Dialogue Models from Limited Personalized Data](https://mlsanity.com/paper/2106.06169/) (2021)
- [Don't Say No: Jailbreaking LLM by Suppressing Refusal](https://mlsanity.com/paper/2404.16369/) (2024)
- [Improving Factual Consistency of Abstractive Summarization via Question Answering](https://mlsanity.com/paper/2105.04623/) (2021)
- [Paragraph-level Simplification of Medical Texts](https://mlsanity.com/paper/2104.05767/) (2021)
- [Contrastive Learning Reduces Hallucination in Conversations](https://mlsanity.com/paper/2212.10400/) (2022)
- [AuGPT: Auxiliary Tasks and Data Augmentation for End-To-End Dialogue with Pre-Trained Language Models](https://mlsanity.com/paper/2102.05126/) (2021)
- [EVA2.0: Investigating Open-Domain Chinese Dialogue Systems with Large-Scale Pre-Training](https://mlsanity.com/paper/2203.09313/) (2022)
- [Faithfulness in Natural Language Generation: A Systematic Survey of Analysis, Evaluation and Optimization Methods](https://mlsanity.com/paper/2203.05227/) (2022)
- [DeliData: A dataset for deliberation in multi-party problem solving](https://mlsanity.com/paper/2108.05271/) (2021)
- [Building a Role Specified Open-Domain Dialogue System Leveraging Large-Scale Language Models](https://mlsanity.com/paper/2205.00176/) (2022)
- [Diversifying Dialog Generation via Adaptive Label Smoothing](https://mlsanity.com/paper/2105.14556/) (2021)
- [Decomposed Mutual Information Estimation for Contrastive Representation Learning](https://mlsanity.com/paper/2106.13401/) (2021)
- [CAT: A Contextualized Conceptualization and Instantiation Framework for Commonsense Reasoning](https://mlsanity.com/paper/2305.04808/) (2023)
- [A Survey on Human Preference Learning for Large Language Models](https://mlsanity.com/paper/2406.11191/) (2024)
- [Hi, my name is Martha: Using names to measure and mitigate bias in generative dialogue models](https://mlsanity.com/paper/2109.03300/) (2021)
- [Am I Me or You? State-of-the-Art Dialogue Models Cannot Maintain an Identity](https://mlsanity.com/paper/2112.05843/) (2021)
- [Recent Trends in Personalized Dialogue Generation: A Review of Datasets, Methodologies, and Evaluations](https://mlsanity.com/paper/2405.17974/) (2024)

## Abstract (arXiv)

Generative dialogue models currently suffer from a number of problems which standard maximum likelihood training does not address. They tend to produce generations that (i) rely too much on copying from the context, (ii) contain repetitions within utterances, (iii) overuse frequent words, and (iv) at a deeper level, contain logical flaws. In this work we show how all of these problems can be addressed by extending the recently introduced unlikelihood loss (Welleck et al., 2019) to these cases. We show that appropriate loss functions which regularize generated outputs to match human distributions are effective for the first three issues. For the last important general issue, we show applying unlikelihood to collected data of what a model should not do is effective for improving logical consistency, potentially paving the way to generative models with greater reasoning ability. We demonstrate the efficacy of our approach across several dialogue tasks.