mlsanityData to 14 Oct 2026

The CRINGE Loss: Learning what language not to model

arXiv 2211.05826 ↗First posted 10 Nov 2022cs.CL cs.AIcited by 39

Not on the map. Why?

Abstract (arXiv)

Standard language model training employs gold human documents or human-human interaction data, and treats all training data as positive examples. Growing evidence shows that even with very large amounts of positive training data, issues remain that can be alleviated with relatively small amounts of negative data -- examples of what the model should not do. In this work, we propose a novel procedure to train with such data called the CRINGE loss (ContRastive Iterative Negative GEneration). We show the effectiveness of this approach across three different experiments on the tasks of safe generation, contradiction avoidance, and open-domain dialogue. Our models outperform multiple strong baselines and are conceptually simple, easy to train and implement.

Key facts. Cited by 39 papers in the map (data through 14 Oct 2026). Builds on 43 of its 50 references that are in the map. First posted on arXiv 10 Nov 2022.

Builds on

43 of 50 references.

References · 43

  1. 2017Attention Is All You Needcited by 60,361
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
  4. 2022Training language models to follow instructions with human feedbackcited by 15,971
  5. 2017Proximal Policy Optimization Algorithmscited by 15,215
  6. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  7. 2017Deep reinforcement learning from human preferencescited by 3,912
  8. 2020Learning to summarize from human feedbackcited by 2,588
  9. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,411
  10. 2019Fine-Tuning Language Models from Human Preferencescited by 2,150
  11. 2021WebGPT: Browser-assisted question-answering with human feedbackcited by 1,745
  12. 2020RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Modelscited by 1,255
31 more
  1. 2020Leveraging Passage Retrieval with Generative Models for Open Domain Question Answeringcited by 1,210
  2. 2021A General Language Assistant as a Laboratory for Alignmentcited by 1,055
  3. 2019Plug and Play Language Models: A Simple Approach to Controlled Text Generationcited by 799
  4. 2020Recipes for building an open-domain chatbotcited by 593
  5. 2019Neural Text Generation with Unlikelihood Trainingcited by 503
  6. 2020GeDi: Generative Discriminator Guided Sequence Generationcited by 378
  7. 2021FUDGE: Controlled Text Generation With Future Discriminatorscited by 376
  8. 2020The Pushshift Reddit Datasetcited by 292
  9. 2021Recursively Summarizing Books with Human Feedbackcited by 290
  10. 2019Way Off-Policy Batch Deep Reinforcement Learning of Implicit Human Preferences in Dialogcited by 284
  11. 2021Beyond Goldfish Memory: Long-Term Open-Domain Conversationcited by 267
  12. 2022Quark: Controllable Text Generation with Reinforced Unlearningcited by 237
  13. 2021Internet-Augmented Dialogue Generationcited by 217
  14. 2016Ex Machina: Personal Attacks Seen at Scalecited by 210
  15. 2022BlenderBot 3: a deployed conversational agent that continually learns to responsibly engagecited by 198
  16. 2019The Second Conversational Intelligence Challenge (ConvAI2)cited by 179
  17. 2019Build it Break it Fix it for Dialogue Safety: Robustness from Adversarial Human Attackcited by 176
  18. 2017ParlAI: A Dialog Research Software Platformcited by 139
  19. 2020Can You Put it All Together: Evaluating Conversational Agents' Ability to Blend Skillscited by 131
  20. 2019Learning from Dialogue after Deployment: Feed Yourself, Chatbot!cited by 102
  21. 2021Anticipating Safety Issues in E2E Conversational AI: Framework and Toolingcited by 78
  22. 2019Better Rewards Yield Better Summaries: Learning to Summarise Without Referencescited by 77
  23. 2020I like fish, especially dolphins: Addressing Contradictions in Dialogue Modelingcited by 68
  24. 2022DIRECTOR: Generator-Classifiers For Supervised Language Modelingcited by 38
  25. 2022Learning New Skills after Deployment: Improving open-domain internet-driven dialogue with human feedbackcited by 36
  26. 2015A Survey of Available Corpora for Building Data-Driven Dialogue Systemscited by 33
  27. 2019Negative Training for Neural Dialogue Response Generationcited by 24
  28. 2021Am I Me or You? State-of-the-Art Dialogue Models Cannot Maintain an Identitycited by 22
  29. 2022A Simple Contrastive Learning Objective for Alleviating Neural Text Degenerationcited by 18
  30. 2022Learning from data in the mixed adversarial non-adversarial case: Finding the helpers and ignoring the trollscited by 11
  31. 2022Jam or Cream First? Modeling Ambiguity in Neural Machine Translation with SCONEScited by 10

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 2; 2023: 10; 2024: 18; 2025: 7; 2026: 2.

39 papers.

Sort:

By area:

  1. 2024Self-Rewarding Language Modelscited by 681
  2. 2024Iterative Reasoning Preference Optimizationcited by 246
  3. 2023Large Language Model Alignment: A Surveycited by 243
  4. 2024A Mechanistic Understanding of Alignment Algorithms: A Case Study on DPO and Toxicitycited by 185
  5. 2024Preference Fine-Tuning of LLMs Should Leverage Suboptimal, On-Policy Datacited by 185
  6. 2024Direct Nash Optimization: Teaching Language Models to Self-Improve with General Preferencescited by 178
  7. 2023Controlled Decoding from Language Modelscited by 135
  8. 2025Advances and Challenges in Foundation Agents: From Brain-Inspired Intelligence to Evolutionary, Collaborative, and Safe Systemscited by 126
  9. 2024On Prompt-Driven Safeguarding for Large Language Modelscited by 116
  10. 2023Some things are more CRINGE than others: Iterative Preference Optimization with the Pairwise Cringe Losscited by 108
  11. 2024LLM Unlearning via Loss Adjustment with Only Forget Datacited by 75
  12. 2024Enhancing Large Vision Language Models with Self-Training on Image Comprehensioncited by 70
27 more
  1. 2023Multi-Party Chat: Conversational Agents in Group Settings with Humans and Modelscited by 54
  2. 2024WPO: Enhancing RLHF with Weighted Preference Optimizationcited by 54
  3. 2024Anchored Preference Optimization and Contrastive Revisions: Addressing Underspecification in Alignmentcited by 46
  4. 2023Click: Controllable Text Generation with Sequence Likelihood Contrastive Learningcited by 41
  5. 2025Exploring Criteria of Loss Reweighting to Enhance LLM Unlearningcited by 39
  6. 2024Whispering Experts: Neural Interventions for Toxicity Mitigation in Language Modelscited by 32
  7. 2024How Far Are We From AGI: Are LLMs All We Need?cited by 31
  8. 2022Reward Gaming in Conditional Text Generationcited by 30
  9. 2024A Comprehensive Survey of Direct Preference Optimization: Datasets, Theories, Variants, and Applicationscited by 27
  10. 2024High-Dimension Human Value Representation in Large Language Modelscited by 20
  11. 2022Controllable Text Generation with Language Constraintscited by 18
  12. 2023Leveraging Implicit Feedback from Deployment Data in Dialoguecited by 14
  13. 2025Reward Model Overoptimisation in Iterated RLHFcited by 12
  14. 2025Boosting Chart-to-Code Generation in MLLM via Dual Preference-Guided Refinementcited by 11
  15. 2023Improving Open Language Models by Learning from Organic Interactionscited by 10
  16. 2023Learn What NOT to Learn: Towards Generative Safety in Chatbotscited by 8
  17. 2023Revisiting the Architectures like Pointer Networks to Efficiently Improve the Next Word Distribution, Summarization Factuality, and Beyondcited by 8
  18. 2024Reinforcement Learning from Reflective Feedback (RLRF): Aligning and Improving LLMs via Fine-Grained Self-Reflectioncited by 8
  19. 2024Forewarned is Forearmed: Leveraging LLMs for Data Synthesis through Failure-Inducing Explorationcited by 7
  20. 2026Explainable LLM Unlearning Through Reasoningcited by 7
  21. 2024Minor DPO reject penalty to increase training robustnesscited by 5
  22. 2023System-Level Natural Language Feedbackcited by 4
  23. 2025Generative Model Unlearning: A Survey through Target Events, Unlearning Operators, and Evaluation Protocolscited by 4
  24. 2025Teaching Models to Understand (but not Generate) High-risk Datacited by 2
  25. 2024Learning from Relevant Subgoals in Successful Dialogs using Iterative Training for Task-oriented Dialog Systemscited by 1
  26. 2025Beautiful Images, Toxic Words: Understanding and Addressing Offensive Text in Generated Imagescited by 0
  27. 2026Forgetting Similar Samples: Can Machine Unlearning Do it Better?cited by 0