Data to 7 Oct 2026

Contrastive Preference Learning: Learning from Human Feedback without RL

arXiv 2310.13639 ↗First posted 20 Oct 2023cs.LG cs.AIcited by 77

On the map: LLM alignment / RLHF / preference optimisation → Reward modeling and RLHF

Abstract (arXiv)

Reinforcement Learning from Human Feedback (RLHF) has emerged as a popular paradigm for aligning models with human intent. Typically RLHF algorithms operate in two phases: first, use human preferences to learn a reward function and second, align the model by optimizing the learned reward via reinforcement learning (RL). This paradigm assumes that human preferences are distributed according to reward, but recent work suggests that they instead follow the regret under the user's optimal policy. Thus, learning a reward function from feedback is not only based on a flawed assumption of human preference, but also leads to unwieldy optimization challenges that stem from policy gradients or bootstrapping in the RL phase. Because of these optimization challenges, contemporary RLHF methods restrict themselves to contextual bandit settings (e.g., as in large language models) or limit observation dimensionality (e.g., state-based robotics). We overcome these limitations by introducing a new family of algorithms for optimizing behavior from human feedback using the regret-based model of human preferences. Using the principle of maximum entropy, we derive Contrastive Preference Learning (CPL), an algorithm for learning optimal policies from preferences without learning reward functions, circumventing the need for RL. CPL is fully off-policy, uses only a simple contrastive objective, and can be applied to arbitrary MDPs. This enables CPL to elegantly scale to high-dimensional and sequential RLHF problems while being simpler than prior methods.

Key facts. Cited by 77 papers in the map (data through 7 Oct 2026). Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 48 of its 63 references that are in the map. First posted on arXiv 20 Oct 2023.

Builds on

48 of 63 references.

Closest ancestors · 13

Same direction, newest first

1 more

2017Deep reinforcement learning from human preferences12 Jun

Other references · 35

2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
2022Training language models to follow instructions with human feedbackcited by 15,712
2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
2018Representation Learning with Contrastive Predictive Codingcited by 7,662
2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,482
2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
2020Learning to summarize from human feedbackcited by 2,562
2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problemscited by 1,873
2020D4RL: Datasets for Deep Data-Driven Reinforcement Learningcited by 1,406
2021Offline Reinforcement Learning with Implicit Q-Learningcited by 1,245
2019Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learningcited by 1,168

23 more

2023Training Diffusion Models with Reinforcement Learningcited by 939
2021What Matters in Learning from Offline Human Demonstrations for Robot Manipulationcited by 699
2017Reinforcement Learning with Deep Energy-Based Policiescited by 659
2020Contrastive Learning with Hard Negative Samplescited by 619
2020Reinforcement Learning with Augmented Datacited by 452
2023Aligning Text-to-Image Models using Human Feedbackcited by 418
2022VIP: Towards Universal Visual Reward and Representation via Value-Implicit Pre-Trainingcited by 384
2020Is Pessimism Provably Efficient for Offline RL?cited by 362
2021Mastering Visual Continuous Control: Improved Data-Augmented Reinforcement Learningcited by 359
2019Extrapolating Beyond Suboptimal Demonstrations via Inverse Reinforcement Learning from Observationscited by 256
2022Contrastive Learning as Goal-Conditioned Reinforcement Learningcited by 241
2018Deep Reinforcement Learning and the Deadly Triadcited by 170
2023Extreme Q-Learning: MaxEnt RL without Entropycited by 113
2022Few-Shot Preference Learning for Human-in-the-Loop RLcited by 85
2023Preference Transformer: Modeling Human Preferences using Transformers for RLcited by 77
2020Safe Imitation Learning via Fast Bayesian Reward Inference from Preferencescited by 73
2018Learning to Extract Coherent Summary via Deep Reinforcement Learningcited by 63
2023Dual RL: Unification and New Methods for Reinforcement and Imitation Learningcited by 48
2021Training Larger Networks for Deep Reinforcement Learningcited by 37
2022A Ranking Game for Imitation Learningcited by 18
2023Distance Weighted Supervised Learning for Offline Interaction Datacited by 18
2021Offline Preference-Based Apprenticeship Learningcited by 17
2019The Green Choice: Learning and Influencing Human Decisions on Shared Roadscited by 14

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 3; 2024: 37; 2025: 25; 2026: 12.

77 papers.

Sort:

By area:

2024Contrastive Preference Optimization: Pushing the Boundaries of LLM Performance in Machine Translationcited by 390
2024A Minimaximalist Approach to Reinforcement Learning from Human Feedbackcited by 141
2024Human Alignment of Large Language Models through Online Preference Optimisationcited by 90
2024Direct Multi-Turn Preference Optimization for Language Agentscited by 63
2024TreeBoN: Enhancing Inference-Time Alignment with Speculative Tree-Search and Best-of-N Samplingcited by 61
2024A Dense Reward View on Aligning Text-to-Image Diffusion with Preferencecited by 55
2024TRANSIC: Sim-to-Real Policy Transfer by Learning from Online Correctioncited by 50
2024(Perhaps) Beyond Human Translation: Harnessing Multi-Agent Collaboration for Translating Ultra-Long Literary Textscited by 49
2024X-ALMA: Plug & Play Modules and Adaptive Rejection for Quality Translation at Scalecited by 36
2024AI Safety in Generative AI Large Language Models: A Surveycited by 31
2024Reward Model Learning vs. Direct Policy Optimization: A Comparative Analysis of Learning from Human Preferencescited by 27
2024Heterogeneous Contrastive Learning for Foundation Models and Beyondcited by 27

38 more

2024Inverse-RLignment: Large Language Model Alignment from Demonstrations through Inverse Reinforcement Learningcited by 27
2023Promptable Behaviors: Personalizing Multi-Objective Rewards from Human Preferencescited by 25
2025It Takes Two: Your GRPO Is Secretly DPOcited by 25
2024Task Success is not Enough: Investigating the Use of Video-Language Models as Behavior Critics for Catching Undesirable Agent Behaviorscited by 23
2024Improving Reinforcement Learning from Human Feedback Using Contrastive Rewardscited by 23
2024On the Essence and Prospect: An Investigation of Alignment Approaches for Big Modelscited by 21
2024Scalable Reinforcement Post-Training Beyond Static Human Prompts: Evolving Alignment via Asymmetric Self-Playcited by 21
2024Understanding Layer Significance in LLM Alignmentcited by 19
2024Humor in AI: Massive Scale Crowd-Sourced Preferences and Benchmarks for Cartoon Captioningcited by 18
2024Towards a Unified View of Preference Learning for Large Language Models: A Surveycited by 17
2024Self-Play with Adversarial Critic: Provable and Scalable Offline Alignment for Language Modelscited by 16
2025Do We Need to Verify Step by Step? Rethinking Process Supervision from a Theoretical Perspectivecited by 16
2024Understanding the Learning Dynamics of Alignment with Human Feedbackcited by 15
2024Listwise Reward Estimation for Offline Preference-based Reinforcement Learningcited by 14
2024Preference Alignment with Flow Matchingcited by 11
2024YODA: Teacher-Student Progressive Learning for Language Modelscited by 10
2024Forward KL Regularized Preference Optimization for Aligning Diffusion Policiescited by 10
2024Regularized Conditional Diffusion Model for Multi-Task Preference Alignmentcited by 9
2024A Preference-driven Paradigm for Enhanced Translation with Large Language Modelscited by 9
2024Batch Active Learning of Reward Functions from Human Preferencescited by 8
2024Can DPO Learn Diverse Human Values? A Theoretical Scaling Lawcited by 7
2024LASeR: Learning to Adaptively Select Reward Models with Multi-Armed Banditscited by 7
2024DeformPAM: Data-Efficient Learning for Long-horizon Deformable Object Manipulation via Preference-based Action Alignmentcited by 7
2025Inverse Reinforcement Learning Meets Large Language Model Post-Training: Basics, Advances, and Opportunitiescited by 7
2023RLHF and IIA: Perverse Incentivescited by 6
2024Triple Preference Optimization: Achieving Better Alignment using a Single Step Optimizationcited by 6
2025Direct Post-Training Preference Alignment for Multi-Agent Motion Generation Models Using Implicit Feedback from Pre-training Demonstrationscited by 6
2025Robot-Gated Interactive Imitation Learning with Adaptive Intervention Mechanismcited by 6
2025Collaborate, Deliberate, Evaluate: How LLM Alignment Affects Coordinated Multi-Agent Outcomescited by 6
2023Rescue: Ranking LLM Responses with Partial Ordering to Improve Response Generationcited by 4
2024Robot Air Hockey: A Manipulation Testbed for Robot Learning with Reinforcement Learningcited by 4
2024ContraSolver: Self-Alignment of Language Models by Resolving Internal Preference Contradictionscited by 4
2025Influencing Humans to Conform to Preference Models for RLHFcited by 4
2025Policy Learning from Large Vision-Language Model Feedback without Reward Modelingcited by 4
2025Mitigating Length Bias in RLHF through a Causal Lenscited by 4
2026UniIntervene: Agentic Intervention for Efficient Real-World Reinforcement Learningcited by 4
2024Dynamic Normativity: Necessary and Sufficient Conditions for Value Alignmentcited by 3
2025Solving the Unsolvable: Translating Case Law in Hong Kongcited by 3

50 most cited of 77 All 77 (JSON)

Similar, not cited

Close in wording, no citation either way