mlsanityData to 7 Oct 2026

SeRA: Self-Reviewing and Alignment of Large Language Models using Implicit Reward Margins

arXiv 2410.09362 ↗First posted 12 Oct 2024cs.LG cs.AIcited by 5

On the map: LLM alignment / RLHF / preference optimisation → LLM preference optimization methods

Abstract (arXiv)

Direct alignment algorithms (DAAs), such as direct preference optimization (DPO), have become popular alternatives for Reinforcement Learning from Human Feedback (RLHF) due to their simplicity, efficiency, and stability. However, the preferences used in DAAs are usually collected before the alignment training begins and remain unchanged (off-policy). This can lead to two problems where the policy model (1) picks up on spurious correlations in the dataset (as opposed to learning the intended alignment expressed in the human preference labels), and (2) overfits to feedback on off-policy trajectories that have less likelihood of being generated by an updated policy model. To address these issues, we introduce Self-Reviewing and Alignment (SeRA), a cost-efficient and effective method that can be readily combined with existing DAAs. SeRA comprises of two components: (1) sample selection using implicit reward margins, which helps alleviate over-fitting to some undesired features, and (2) preference bootstrapping using implicit rewards to augment preference data with updated policy models in a cost-efficient manner. Extensive experimentation, including some on instruction-following tasks, demonstrate the effectiveness and generality of SeRA in training LLMs on offline preference datasets with DAAs.

Key facts. Cited by 5 papers in the map (data through 7 Oct 2026). Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 45 of its 88 references that are in the map. First posted on arXiv 12 Oct 2024.

Builds on

45 of 88 references.

Closest ancestors · 14

Same direction, newest first

  1. 2024WPO: Enhancing RLHF with Weighted Preference Optimization17 Jun
  2. 2024Bootstrapping Language Models with DPO Implicit Rewards14 Jun
  3. 2024Spread Preference Annotation: Direct Preference Judgment for Efficient LLM Alignment6 Jun
  4. 2024SimPO: Simple Preference Optimization with a Reference-Free Reward23 May
  5. 2024sDPO: Don't Use Your Data All at Once28 Mar
  6. 2024Disentangling Length from Quality in Direct Preference Optimization28 Mar
  7. 2024ORPO: Monolithic Preference Optimization without Reference Model12 Mar
  8. 2024Curry-DPO: Enhancing Alignment using Curriculum Learning & Ranked Preferences12 Mar
  9. 2024Provably Robust DPO: Aligning Language Models with Noisy Feedback1 Mar
  10. 2024Direct Language Model Alignment from Online AI Feedback7 Feb
  11. 2024DeAL: Decoding-time Alignment for Large Language Models5 Feb
  12. 2024KTO: Model Alignment as Prospect Theoretic Optimization2 Feb
2 more
  1. 2023Adversarial Preference Optimization: Enhancing Your Alignment via RM-LLM Game14 Nov
  2. 2023Zephyr: Direct Distillation of LM Alignment25 Oct

Other references · 31

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2017Proximal Policy Optimization Algorithmscited by 14,441
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  4. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  5. 2023Mistral 7Bcited by 3,065
  6. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 2,950
  7. 2024Mixtral of Expertscited by 1,785
  8. 2023WizardLM: Empowering large pre-trained language models to follow complex instructionscited by 1,116
  9. 2024Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluatorscited by 865
  10. 2023Gemini: A Family of Highly Capable Multimodal Modelscited by 810
  11. 2022Red Teaming Language Models to Reduce Harms: Methods, Scaling Behaviors, and Lessons Learnedcited by 702
  12. 2024Self-Rewarding Language Modelscited by 654
19 more
  1. 2024LLM Evaluators Recognize and Favor Their Own Generationscited by 651
  2. 2023RLAIF vs. RLHF: Scaling Reinforcement Learning from Human Feedback with AI Feedbackcited by 615
  3. 2023Textbooks Are All You Need II: phi-1.5 technical reportcited by 559
  4. 2024Self-Play Fine-Tuning Converts Weak Language Models to Strong Language Modelscited by 552
  5. 2024RewardBench: Evaluating Reward Models for Language Modelingcited by 480
  6. 2023Weak-to-Strong Generalization: Eliciting Strong Capabilities With Weak Supervisioncited by 445
  7. 2023SLiC-HF: Sequence Likelihood Calibration with Human Feedbackcited by 395
  8. 2022Large Language Models are Better Reasoners with Self-Verificationcited by 304
  9. 2024Large Language Models for Mathematical Reasoning: Progresses and Challengescited by 283
  10. 2024Preference Fine-Tuning of LLMs Should Leverage Suboptimal, On-Policy Datacited by 184
  11. 2024Direct Nash Optimization: Teaching Language Models to Self-Improve with General Preferencescited by 176
  12. 2024Secrets of RLHF in Large Language Models Part II: Reward Modelingcited by 170
  13. 2023Benchmarking Cognitive Biases in Large Language Models as Evaluatorscited by 147
  14. 2024DistiLLM: Towards Streamlined Distillation for Large Language Modelscited by 145
  15. 2022Why Exposure Bias Matters: An Imitation Learning Perspective of Error Accumulation in Language Generationcited by 106
  16. 2023Style Over Substance: Evaluation Biases for Large Language Modelscited by 68
  17. 2024Weaver: Foundation Models for Creative Writingcited by 25
  18. 2024Spontaneous Reward Hacking in Iterative Self-Refinementcited by 24
  19. 2023Fine tuning Pre trained Models for Robustness Under Noisy Labelscited by 8

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 4; 2026: 1.

5 papers.

  1. 2025A Survey of Direct Preference Optimizationcited by 35
  2. 2025Implicit Cross-Lingual Rewarding for Efficient Multilingual Preference Alignmentcited by 9
  3. 2025REWARD CONSISTENCY: Improving Multi-Objective Alignment from a Data-Centric Perspectivecited by 5
  4. 2025Alignment through Meta-Weighted Online Sampling: Bridging the Gap between Data Generation and Preference Optimizationcited by 3
  5. 2026Cleansing the Artificial Mind: A Self-Reflective Detoxification Framework for Large Language Modelscited by 1

Similar, not cited

Close in wording, no citation either way

  1. 2025Towards Bridging the Reward-Generation Gap in Direct Alignment Algorithms
  2. 2025AlphaPO: Reward Shape Matters for LLM Alignment
  3. 2024RS-DPO: A Hybrid Rejection Sampling and Direct Preference Optimization Method for Alignment of Large Language Models
  4. 2024Reward-Augmented Data Enhances Direct Preference Alignment of LLMs
  5. 2026Towards Bridging the Gap Between Offline and Iterative Alignment via Preference Distillation
  6. 2024AlphaDPO: Adaptive Reward Margin for Direct Preference Optimization
  7. 2025Mitigating Reward Over-optimization in Direct Alignment Algorithms with Importance Sampling
  8. 2024Human Alignment of Large Language Models through Online Preference Optimisation