mlsanityData to 7 Oct 2026

Direct Large Language Model Alignment Through Self-Rewarding Contrastive Prompt Distillation

arXiv 2402.11907 ↗First posted 19 Feb 2024cs.CLcited by 18

On the map: LLM reasoning → Reinforcement learning for reasoning

also in 2 more
Abstract (arXiv)

Aligning large language models (LLMs) with human expectations without human-annotated preference data is an important problem. In this paper, we propose a method to evaluate the response preference by using the output probabilities of response pairs under contrastive prompt pairs, which could achieve better performance on LLaMA2-7B and LLaMA2-13B compared to RLAIF. Based on this, we propose an automatic alignment method, Direct Large Model Alignment (DLMA). First, we use contrastive prompt pairs to automatically generate preference data. Then, we continue to evaluate the generated preference data using contrastive prompt pairs and calculate a self-rewarding score. Finally, we use the DPO algorithm to effectively align LLMs by combining this self-rewarding score. In the experimental stage, our DLMA method could surpass the textttRLHF method without relying on human-annotated preference data.

Key facts. Cited by 18 papers in the map (data through 7 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 21 of its 38 references that are in the map. First posted on arXiv 19 Feb 2024.

Builds on

21 of 38 references.

Closest ancestors · 3

Same direction, newest first

  1. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Model29 May
  2. 2022Constitutional AI: Harmlessness from AI Feedback15 Dec
  3. 2017Proximal Policy Optimization Algorithms20 Jul

Other references · 18

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  3. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  4. 2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,579
  5. 2023Jailbroken: How Does LLM Safety Training Fail?cited by 1,565
  6. 2021A General Language Assistant as a Laboratory for Alignmentcited by 1,023
  7. 2023BeaverTails: Towards Improved Safety Alignment of LLM via a Human-Preference Datasetcited by 828
  8. 2023Safe RLHF: Safe Reinforcement Learning from Human Feedbackcited by 630
  9. 2023RLAIF vs. RLHF: Scaling Reinforcement Learning from Human Feedback with AI Feedbackcited by 615
  10. 2023The Falcon Series of Open Language Modelscited by 526
  11. 2023Weak-to-Strong Generalization: Eliciting Strong Capabilities With Weak Supervisioncited by 445
  12. 2023How Good Are GPT Models at Machine Translation? A Comprehensive Evaluationcited by 344
6 more
  1. 2023Principle-Driven Self-Alignment of Language Models from Scratch with Minimal Human Supervisioncited by 318
  2. 2023Preference Ranking Optimization for Human Alignmentcited by 283
  3. 2023LEVER: Learning to Verify Language-to-Code Generation with Executioncited by 228
  4. 2023LLM Self Defense: By Self Examination, LLMs Know They Are Being Trickedcited by 226
  5. 2023Can LLM-Generated Misinformation Be Detected?cited by 173
  6. 2024MLLM-Protector: Ensuring MLLM's Safety without Hurting Performancecited by 111

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 8; 2025: 9; 2026: 1.

18 papers.

Sort:

By area:

  1. 2025Interactive Post-Training for Vision-Language-Action Modelscited by 129
  2. 2024A Survey on Self-Evolution of Large Language Modelscited by 86
  3. 2024TIS-DPO: Token-level Importance Sampling for Direct Preference Optimization With Estimated Weightscited by 42
  4. 2024A safety realignment framework via subspace-oriented model fusion for large language modelscited by 40
  5. 2024Recent Advances of Multimodal Continual Learning: A Comprehensive Surveycited by 39
  6. 2025A Survey of Direct Preference Optimizationcited by 35
  7. 2025LongPO: Long Context Self-Evolution of Large Language Models through Short-to-Long Preference Optimizationcited by 24
  8. 2025Omni-SafetyBench: A Benchmark for Safety Evaluation of Audio-Visual Large Language Modelscited by 13
  9. 2025MuSC: Improving Complex Instruction Following with Multi-granularity Self-Contrastive Trainingcited by 12
  10. 2025Can LLMs Explain Themselves Counterfactually?cited by 10
  11. 2024Maximizing Alignment with Minimal Feedback: Efficiently Learning Rewards for Visuomotor Robot Policy Alignmentcited by 6
  12. 2025Direct Post-Training Preference Alignment for Multi-Agent Motion Generation Models Using Implicit Feedback from Pre-training Demonstrationscited by 6
6 more
  1. 2024Data-Centric Human Preference with Rationales for Direct Preference Alignmentcited by 4
  2. 2024On the Transformations across Reward Model, Parameter Update, and In-Context Promptcited by 1
  3. 2024Just Say What You Want: Only-prompting Self-rewarding Online Preference Optimizationcited by 1
  4. 2025How well can LLMs provide planning feedback in grounded environments?cited by 1
  5. 2025Factors That Support Grounded Responses in LLM Conversations: A Rapid Reviewcited by 0
  6. 2026Beyond Tokens: A Survey on Decoding Methods for Large Language and Vision-Language Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Dynamic Rewarding with Prompt Optimization Enables Tuning-free Self-Alignment of Language Models
  2. 2025ComPO: Preference Alignment via Comparison Oracles
  3. 2025Robust LLM Alignment via Distributionally Robust Direct Preference Optimization
  4. 2025Align-Pro: A Principled Approach to Prompt Optimization for LLM Alignment
  5. 2024Noise Contrastive Alignment of Language Models with Explicit Rewards
  6. 2024REAL: Response Embedding-based Alignment for LLMs
  7. 2024Cal-DPO: Calibrated Direct Preference Optimization for Language Model Alignment
  8. 2025Reward-aware Preference Optimization: A Unified Mathematical Framework for Model Alignment