mlsanityData to 7 Oct 2026

RLCD: Reinforcement Learning from Contrastive Distillation for Language Model Alignment

arXiv 2307.12950 ↗First posted 24 Jul 2023cs.CL cs.AIcited by 42

On the map: LLM alignment / RLHF / preference optimisation → LLM preference optimization methods

Abstract (arXiv)

We propose Reinforcement Learning from Contrastive Distillation (RLCD), a method for aligning language models to follow principles expressed in natural language (e.g., to be more harmless) without using human feedback. RLCD creates preference pairs from two contrasting model outputs, one using a positive prompt designed to encourage following the given principles, and one using a negative prompt designed to encourage violating them. Using two different prompts causes model outputs to be more differentiated on average, resulting in cleaner preference labels in the absence of human annotations. We then use the preference pairs to train a preference model, which is in turn used to improve a base unaligned language model via reinforcement learning. Empirically, RLCD outperforms RLAIF (Bai et al., 2022b) and context distillation (Huang et al., 2022) baselines across three diverse alignment tasks--harmlessness, helpfulness, and story outline generation--and when using both 7B and 30B model scales for simulating preference data.

Key facts. Cited by 42 papers in the map (data through 7 Oct 2026). Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 26 of its 41 references that are in the map. First posted on arXiv 24 Jul 2023.

Builds on

26 of 41 references.

Closest ancestors · 3

Same direction, newest first

  1. 2023Principle-Driven Self-Alignment of Language Models from Scratch with Minimal Human Supervision4 May
  2. 2022Training language models to follow instructions with human feedback4 Mar
  3. 2021A General Language Assistant as a Laboratory for Alignment1 Dec

Other references · 23

  1. 2017Proximal Policy Optimization Algorithmscited by 14,441
  2. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  4. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  5. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  6. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  7. 2023Lost in the Middle: How Language Models Use Long Contextscited by 3,383
  8. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 2,950
  9. 2020Learning to summarize from human feedbackcited by 2,562
  10. 2016Sequence-Level Knowledge Distillationcited by 761
  11. 2020CURL: Contrastive Unsupervised Representations for Reinforcement Learningcited by 745
  12. 2023AlpacaFarm: A Simulation Framework for Methods that Learn from Human Feedbackcited by 738
11 more
  1. 2023Extending Context Window of Large Language Models via Positional Interpolationcited by 721
  2. 2022Large Language Models Can Self-Improvecited by 656
  3. 2023Principled Reinforcement Learning with Human Feedback from Pairwise or K-wise Comparisonscited by 244
  4. 2022Contrastive Learning as Goal-Conditioned Reinforcement Learningcited by 241
  5. 2021Challenges in Detoxifying Language Modelscited by 169
  6. 2022Learning by Distilling Contextcited by 128
  7. 2022CIC: Contrastive Intrinsic Control for Unsupervised Skill Discoverycited by 66
  8. 2023Learning to Generate Better Than Your LLMcited by 49
  9. 2021Return-Based Contrastive Representation Learning for Reinforcement Learningcited by 36
  10. 2022Understanding Deep Contrastive Learning via Coordinate-wise Optimizationcited by 35
  11. 2022Prompt Injection: Parameterization of Fixed Inputscited by 35

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 7; 2024: 19; 2025: 12; 2026: 4.

42 papers.

Sort:

By area:

  1. 2023A Long Way to Go: Investigating Length Correlations in RLHFcited by 285
  2. 2024Is DPO Superior to PPO for LLM Alignment? A Comprehensive Studycited by 262
  3. 2024V-STaR: Training Verifiers for Self-Taught Reasonerscited by 214
  4. 2024Direct Nash Optimization: Teaching Language Models to Self-Improve with General Preferencescited by 176
  5. 2023Some things are more CRINGE than others: Iterative Preference Optimization with the Pairwise Cringe Losscited by 108
  6. 2023SALMON: Self-Alignment with Instructable Reward Modelscited by 54
  7. 2024Language Models Resist Alignment: Evidence From Data Compressioncited by 50
  8. 2024Self-Alignment of Large Language Models via Monopolylogue-based Social Scene Simulationcited by 46
  9. 2023Gaining Wisdom from Setbacks: Aligning Large Language Models via Mistake Analysiscited by 34
  10. 2024Surveying the Effects of Quality, Diversity, and Complexity in Synthetic Data From Large Language Modelscited by 31
  11. 2025Curriculum-RLAIF: Curriculum Alignment with Reinforcement Learning from AI Feedbackcited by 30
  12. 2024West-of-N: Synthetic Preferences for Self-Improving Reward Modelscited by 27
30 more
  1. 2025From Perception to Cognition: A Survey of Vision-Language Interactive Reasoning in Multimodal Large Language Modelscited by 26
  2. 2024Weaver: Foundation Models for Creative Writingcited by 25
  3. 2023Peering Through Preferences: Unraveling Feedback Acquisition for Aligning Large Language Modelscited by 23
  4. 2024MACPO: Weak-to-Strong Alignment via Multi-Agent Contrastive Preference Optimizationcited by 20
  5. 2024Humor in AI: Massive Scale Crowd-Sourced Preferences and Benchmarks for Cartoon Captioningcited by 18
  6. 2024Training Language Models to Critique With Multi-agent Feedbackcited by 16
  7. 2023Reasons to Reject? Aligning Language Models with Judgmentscited by 15
  8. 2024Seeing Eye to AI: Human Alignment via Gaze-Based Response Rewards for Large Language Modelscited by 11
  9. 2025Recontextualization Mitigates Specification Gaming without Modifying the Specificationcited by 11
  10. 2024Mixture of insighTful Experts (MoTE): The Synergy of Thought Chains and Expert Mixtures in Self-Alignmentcited by 10
  11. 2024Understanding Synthetic Context Extension via Retrieval Headscited by 10
  12. 2025GLaD: Geometric Latent Distillation for Vision-Language-Action Modelscited by 10
  13. 2024Boosting Reward Model with Preference-Conditional Multi-Aspect Synthetic Data Generationcited by 9
  14. 2026Evaluating Decision Models for Text Annotation in Computational Social Sciencecited by 8
  15. 2024Maximizing Alignment with Minimal Feedback: Efficiently Learning Rewards for Visuomotor Robot Policy Alignmentcited by 6
  16. 2025Debiasing Reward Models by Representation Learning with Guaranteescited by 6
  17. 2024Exploring Failure Cases in Multimodal Reasoning About Physical Dynamicscited by 5
  18. 2023ZYN: Zero-Shot Reward Models with Yes-No Questions for RLAIFcited by 4
  19. 2024Jailbreak Instruction-Tuned LLMs via end-of-sentence MLP Re-weightingcited by 4
  20. 2025Making VLMs More Robot-Friendly: Self-Critical Distillation of Low-Level Procedural Reasoningcited by 4
  21. 2025Learning What to Trust: Bayesian Prior-Guided Optimization for Visual Generationcited by 4
  22. 2026Just Ask Jev: Reinforcement Learning for Calibrated Decisions as a Zero-Shot Detector of AI Alignment Failurescited by 4
  23. 2025OASST-ETC Dataset: Alignment Signals from Eye-tracking Analysis of LLM Responsescited by 3
  24. 2024PopAlign: Diversifying Contrasting Patterns for a More Comprehensive Alignmentcited by 2
  25. 2025Probability-Consistent Preference Optimization for Enhanced LLM Reasoningcited by 2
  26. 2025PersRM-R1: Enhance Personalized Reward Modeling with Reinforcement Learningcited by 2
  27. 2025Leveraging Importance Sampling to Detach Alignment Modules from Large Language Modelscited by 1
  28. 2025Beyond Direct Generation: A Decomposed Approach to Well-Crafted Screenwriting with LLMscited by 1
  29. 2026Reward Learning from Best-of-N Preference Data: Targets, Tradeoffs, and Design Principlescited by 0
  30. 2026OmniMed-Jev: Calibrating LVLM Confidence for Trustworthy Medical Multimodal Decisions via System Onecited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025AlignDistil: Token-Level Language Model Alignment as Adaptive Policy Distillation
  2. 2024Contrastive Policy Gradient: Aligning LLMs on sequence-level scores in a supervised-friendly fashion
  3. 2022Is Reinforcement Learning (Not) for Natural Language Processing: Benchmarks, Baselines, and Building Blocks for Natural Language Policy Optimization
  4. 2025Beyond RLHF: A Unified Theoretical Framework of Alignment
  5. 2025Doubly Robust Alignment for Large Language Models
  6. 2023Leftover Lunch: Advantage-based Offline Reinforcement Learning for Language Models
  7. 2024Inverse-RLignment: Large Language Model Alignment from Demonstrations through Inverse Reinforcement Learning
  8. 2024Reinforcement Learning for Aligning Large Language Models Agents with Interactive Environments: Quantifying and Mitigating Prompt Overfitting