Data to 7 Oct 2026

Some things are more CRINGE than others: Iterative Preference Optimization with the Pairwise Cringe Loss

arXiv 2312.16682 ↗First posted 27 Dec 2023cs.CL cs.AIcited by 108

On the map: LLM alignment / RLHF / preference optimisation → LLM preference optimization methods

Abstract (arXiv)

Practitioners commonly align large language models using pairwise preferences, i.e., given labels of the type response A is preferred to response B for a given input. Perhaps less commonly, methods have also been developed for binary feedback, i.e. training models given labels of type response A is good or bad. We show how an existing performant binary feedback method, the Cringe Loss (Adolphs et al., 2022), can be generalized to the pairwise preference setting using a simple soft margin extension. Pairwise Cringe Loss is straightforward to implement and efficient to train, and we find it outperforms state-of-the-art preference optimization algorithms such as PPO and DPO on the AlpacaFarm benchmark. We show that iterations of training of our model are important for improved results, and that we can generalize DPO to Iterative DPO in the same way.

Key facts. Cited by 108 papers in the map (data through 7 Oct 2026). Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 25 of its 39 references that are in the map. First posted on arXiv 27 Dec 2023.

Builds on

25 of 39 references.

Closest ancestors · 4

Same direction, newest first

Other references · 21

2017Proximal Policy Optimization Algorithmscited by 14,441
2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
2014Sequence to Sequence Learning with Neural Networkscited by 2,974
2019The Curious Case of Neural Text Degenerationcited by 2,881
2020Learning to summarize from human feedbackcited by 2,562
2020Don't Stop Pretraining: Adapt Language Models to Domains and Taskscited by 1,548
2023AlpacaFarm: A Simulation Framework for Methods that Learn from Human Feedbackcited by 738
2020Recipes for building an open-domain chatbotcited by 593
2019Neural Text Generation with Unlikelihood Trainingcited by 489
2023Reinforced Self-Training (ReST) for Language Modelingcited by 486
2023RRHF: Rank Responses to Align Language Models with Human Feedback without tearscited by 454

9 more

2023SLiC-HF: Sequence Likelihood Calibration with Human Feedbackcited by 395
2021BASE Layers: Simplifying Training of Large, Sparse Modelscited by 315
2022BlenderBot 3: a deployed conversational agent that continually learns to responsibly engagecited by 197
2019Don't Say That! Making Inconsistent Dialogue Unlikely with Unlikelihood Trainingcited by 110
2023Click: Controllable Text Generation with Sequence Likelihood Contrastive Learningcited by 41
2022The CRINGE Loss: Learning what language not to modelcited by 39
2022DIRECTOR: Generator-Classifiers For Supervised Language Modelingcited by 38
2022A Simple Contrastive Learning Objective for Alleviating Neural Text Degenerationcited by 18
2023Improving Open Language Models by Learning from Organic Interactionscited by 10

Built on by

Cited by year

Citing papers per year: 2024: 70; 2025: 38.

108 papers.

Sort:

By area:

2024SimPO: Simple Preference Optimization with a Reference-Free Rewardcited by 951
2024Self-Rewarding Language Modelscited by 654
2024Self-Play Fine-Tuning Converts Weak Language Models to Strong Language Modelscited by 552
2024Interpretable Preferences via Multi-Objective Reward Modeling and Mixture-of-Expertscited by 369
2024RLHF Workflow: From Reward Modeling to Online RLHFcited by 265
2024Direct Language Model Alignment from Online AI Feedbackcited by 257
2024Iterative Reasoning Preference Optimizationcited by 244
2024Self-Play Preference Optimization for Language Model Alignmentcited by 228
2024Assessing the Brittleness of Safety Alignment via Pruning and Low-Rank Modificationscited by 220
2024Meta-Rewarding Language Models: Self-Improving Alignment with LLM-as-a-Meta-Judgecited by 202
2024A Mechanistic Understanding of Alignment Algorithms: A Case Study on DPO and Toxicitycited by 179
2024Direct Nash Optimization: Teaching Language Models to Self-Improve with General Preferencescited by 176

38 more

2025A Minimalist Approach to LLM Reasoning: from Rejection Sampling to Reinforcecited by 141
2025Advances and Challenges in Foundation Agents: From Brain-Inspired Intelligence to Evolutionary, Collaborative, and Safe Systemscited by 125
2025SWEET-RL: Training Multi-Turn LLM Agents on Collaborative Reasoning Taskscited by 108
2024Exploratory Preference Optimization: Harnessing Implicit Q*-Approximation for Sample-Efficient RLHFcited by 106
2024Easy-to-Hard Generalization: Scalable Alignment Beyond Human Supervisioncited by 95
2024mDPO: Conditional Preference Optimization for Multimodal Large Language Modelscited by 92
2024Enhancing Large Vision Language Models with Self-Training on Image Comprehensioncited by 70
2024MAP-Neo: Highly Capable and Transparent Bilingual Large Language Model Seriescited by 67
2024Building Math Agents with Multi-Turn Iterative Preference Learningcited by 66
2024Self-Exploring Language Models: Active Preference Elicitation for Online Alignmentcited by 63
2024Aligning Large Language Models with Representation Editing: A Control Perspectivecited by 62
2024Learning to Clarify: Multi-turn Conversations with Action-Based Contrastive Self-Trainingcited by 53
2024WPO: Enhancing RLHF with Weighted Preference Optimizationcited by 53
2024Self-Consistency Preference Optimizationcited by 50
2024CLIP-DPO: Vision-Language Models as a Source of Preference for Fixing Hallucinations in LVLMscited by 49
2025Diverse Preference Optimizationcited by 48
2024Curry-DPO: Enhancing Alignment using Curriculum Learning & Ranked Preferencescited by 39
2024Discovering Preference Optimization Algorithms with and for Large Language Modelscited by 38
2024Entropy-Regularized Process Reward Modelcited by 38
2024Beyond Scalar Reward Model: Learning Generative Judge from Preference Datacited by 30
2025On the Design of KL-Regularized Policy Gradient Algorithms for LLM Reasoningcited by 30
2025Fine-tuning Large Language Models for Improving Factuality in Legal Question Answeringcited by 28
2025StepWiser: Stepwise Generative Judges for Wiser Reasoningcited by 28
2024West-of-N: Synthetic Preferences for Self-Improving Reward Modelscited by 27
2024Spread Preference Annotation: Direct Preference Judgment for Efficient LLM Alignmentcited by 27
2024On the Limited Generalization Capability of the Implicit Reward Model Induced by Direct Preference Optimizationcited by 26
2025Reinforce LLM Reasoning through Multi-Agent Reflectioncited by 26
2025Bridging Offline and Online Reinforcement Learning for LLMscited by 25
2024Regularized Best-of-N Sampling with Minimum Bayes Risk Objective for Language Model Alignmentcited by 24
2024Preference Tuning with Human Feedback on Language, Speech, and Vision Tasks: A Surveycited by 23
2025BOLT: Bootstrap Long Chain-of-Thought in Language Models without Distillationcited by 23
2025Octopus: Alleviating Hallucination via Dynamic Contrastive Decodingcited by 23
2024Iterative Length-Regularized Direct Preference Optimization: A Case Study on Improving 7B Language Models to GPT-4 Levelcited by 22
2024Understanding Reference Policies in Direct Preference Optimizationcited by 22
2025Step-KTO: Optimizing Mathematical Reasoning through Stepwise Binary Feedbackcited by 22
2025Direct Discriminative Optimization: Your Likelihood-Based Visual Generative Model is Secretly a GAN Discriminatorcited by 22
2024Preference Alignment Improves Language Model-Based TTScited by 21
2025Robust Multi-Objective Preference Alignment with Online DPOcited by 21

50 most cited of 108 All 108 (JSON)

Similar, not cited

Close in wording, no citation either way