mlsanityData to 14 Oct 2026

A Survey on Progress in LLM Alignment from the Perspective of Reward Design

arXiv 2505.02666 ↗First posted 5 May 2025cs.CLcited by 15

On the map: LLM alignment / RLHF / preference optimisation → Reward modeling and RLHF

Abstract (arXiv)

Reward design plays a pivotal role in aligning large language models (LLMs) with human values, serving as the bridge between feedback signals and model optimization. This survey provides a structured organization of reward modeling and addresses three key aspects: mathematical formulation, construction practices, and interaction with optimization paradigms. Building on this, it develops a macro-level taxonomy that characterizes reward mechanisms along complementary dimensions, thereby offering both conceptual clarity and practical guidance for alignment research. The progression of LLM alignment can be understood as a continuous refinement of reward design strategies, with recent developments highlighting paradigm shifts from reinforcement learning (RL)-based to RL-free optimization and from single-task to multi-objective and complex settings.

Key facts. Cited by 15 papers in the map (data through 14 Oct 2026). Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 116 of its 181 references that are in the map. First posted on arXiv 5 May 2025.

Builds on

116 of 181 references.

Closest ancestors · 38

Same direction, newest first

  1. 2025A Comprehensive Survey of Reward Models: Taxonomy, Applications, Challenges, and Future12 Apr
  2. 2025Inverse Reinforcement Learning with Dynamic Reward Scaling for LLM Alignment23 Mar
  3. 2025Sentence-level Reward Model can Generalize Better for Aligning LLM from Human Preference1 Mar
  4. 2025Agentic Reward Modeling: Integrating Human Preferences with Verifiable Correctness Signals for Reliable Reward Systems26 Feb
  5. 2024Approximated Variational Bayesian Inverse Reinforcement Learning for Large Language Model Alignment14 Nov
  6. 2024Optimal Design for Reward Modeling in RLHF22 Oct
  7. 2024Generative Reward Models2 Oct
  8. 2024TLCR: Token-Level Continuous Reward for Fine-grained Reinforcement Learning from Human Feedback23 Jul
  9. 2024Reinforcement Learning for LLM Post-Training: A Survey23 Jul
  10. 2024HAF-RM: A Hybrid Alignment Framework for Reward Model Training4 Jul
  11. 2024Interpretable Preferences via Multi-Objective Reward Modeling and Mixture-of-Experts18 Jun
  12. 2024RLHF Workflow: From Reward Modeling to Online RLHF13 May
26 more
  1. 2024DPO Meets PPO: Reinforced Token Optimization for RLHF29 Apr
  2. 2024Improving Reinforcement Learning from Human Feedback Using Contrastive Rewards12 Mar
  3. 2024ALaRM: Align Language Models via Hierarchical Rewards Modeling11 Mar
  4. 2024Generalizing Reward Modeling for Out-of-Distribution Preference Learning22 Feb
  5. 2024Reward Generalization in RLHF: A Topological Perspective15 Feb
  6. 2024Dense Reward for Free in Reinforcement Learning from Human Feedback1 Feb
  7. 2024Self-Rewarding Language Models18 Jan
  8. 2024Beyond Sparse Rewards: Enhancing Reinforcement Learning with Language Model Critique in Text Generation14 Jan
  9. 2023Improving Generalization of Alignment with Human Preferences through Group Invariant Learning18 Oct
  10. 2023ReMax: A Simple, Effective, and Efficient Reinforcement Learning Method for Aligning Large Language Models16 Oct
  11. 2023SteerLM: Attribute Conditioned SFT as an (User-Steerable) Alternative to RLHF9 Oct
  12. 2023Confronting Reward Model Overoptimization with Constrained RLHF6 Oct
  13. 2023Reward Model Ensembles Help Mitigate Overoptimization4 Oct
  14. 2023Tool-Augmented Reward Modeling2 Oct
  15. 2023Pairwise Proximal Policy Optimization: Harnessing Relative Feedback for LLM Alignment30 Sep
  16. 2023Efficient RLHF: Reducing the Memory Usage of PPO1 Sep
  17. 2023Secrets of RLHF in Large Language Models Part I: PPO11 Jul
  18. 2023Rewarded soups: towards Pareto-optimal alignment by interpolating weights fine-tuned on diverse rewards7 Jun
  19. 2023Fine-Grained Human Feedback Gives Better Rewards for Language Model Training2 Jun
  20. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Model29 May
  21. 2023RRHF: Rank Responses to Align Language Models with Human Feedback without tears11 Apr
  22. 2022Scaling Laws for Reward Model Overoptimization19 Oct
  23. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback12 Apr
  24. 2021WebGPT: Browser-assisted question-answering with human feedback17 Dec
  25. 2019Fine-Tuning Language Models from Human Preferences18 Sep
  26. 2017Deep reinforcement learning from human preferences12 Jun

Other references · 78

  1. 2017Attention Is All You Needcited by 60,361
  2. 2023GPT-4 Technical Reportcited by 18,396
  3. 2022Training language models to follow instructions with human feedbackcited by 15,971
  4. 2017Proximal Policy Optimization Algorithmscited by 15,215
  5. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  6. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  7. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,695
  8. 2021On the Opportunities and Risks of Foundation Modelscited by 4,033
  9. 2023Let's Verify Step by Stepcited by 3,789
  10. 2023Lost in the Middle: How Language Models Use Long Contextscited by 3,455
  11. 2023Self-Refine: Iterative Refinement with Self-Feedbackcited by 3,442
  12. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 3,050
38 more
  1. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,724
  2. 2022Survey of Hallucination in Natural Language Generationcited by 2,075
  3. 2016Concrete Problems in AI Safetycited by 1,642
  4. 2023Not what you've signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injectioncited by 1,443
  5. 2020RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Modelscited by 1,255
  6. 2023LIMA: Less Is More for Alignmentcited by 1,100
  7. 2021A General Language Assistant as a Laboratory for Alignmentcited by 1,055
  8. 2022Large Language Models Are Human-Level Prompt Engineerscited by 1,045
  9. 2024SimPO: Simple Preference Optimization with a Reference-Free Rewardcited by 963
  10. 2023A General Theoretical Paradigm to Understand Learning from Human Preferencescited by 912
  11. 2023CRITIC: Large Language Models Can Self-Correct with Tool-Interactive Critiquingcited by 783
  12. 2023Extracting Training Data from Diffusion Modelscited by 732
  13. 2023Aligning Large Multimodal Models with Factually Augmented RLHFcited by 712
  14. 2021Multimodal Few-Shot Learning with Frozen Language Modelscited by 661
  15. 2023Safe RLHF: Safe Reinforcement Learning from Human Feedbackcited by 656
  16. 2024ORPO: Monolithic Preference Optimization without Reference Modelcited by 562
  17. 2024Self-Play Fine-Tuning Converts Weak Language Models to Strong Language Modelscited by 557
  18. 2024Generative Verifiers: Reward Modeling as Next-Token Predictioncited by 480
  19. 2020Artificial Intelligence, Values and Alignmentcited by 448
  20. 2022Unnatural Instructions: Tuning Language Models with (Almost) No Human Laborcited by 397
  21. 2019Way Off-Policy Batch Deep Reinforcement Learning of Implicit Human Preferences in Dialogcited by 284
  22. 2024Step-DPO: Step-wise Preference Optimization for Long-chain Reasoning of LLMscited by 273
  23. 2024Is DPO Superior to PPO for LLM Alignment? A Comprehensive Studycited by 262
  24. 2022Is Reinforcement Learning (Not) for Natural Language Processing: Benchmarks, Baselines, and Building Blocks for Natural Language Policy Optimizationcited by 246
  25. 2024Iterative Reasoning Preference Optimizationcited by 246
  26. 2023Large Language Model Alignment: A Surveycited by 243
  27. 2022Defining and Characterizing Reward Hackingcited by 203
  28. 2021Societal Biases in Language Generation: Progress and Challengescited by 202
  29. 2024Disentangling Length from Quality in Direct Preference Optimizationcited by 196
  30. 2024Large Language Models for Education: A Survey and Outlookcited by 188
  31. 2020Implementation Matters in Deep Policy Gradients: A Case Study on PPO and TRPOcited by 186
  32. 2023Compositional Exemplars for In-context Learningcited by 144
  33. 2024Token-level Direct Preference Optimizationcited by 135
  34. 2023Beyond One-Preference-Fits-All Alignment: Multi-Objective Direct Preference Optimizationcited by 124
  35. 2022MultiInstruct: Improving Multi-Modal Zero-Shot Learning via Instruction Tuningcited by 109
  36. 2024Rule Based Rewards for Language Model Safetycited by 108
  37. 2023RL4F: Generating Natural Language Feedback with Reinforcement Learning for Repairing Model Outputscited by 105
  38. 2023Shepherd: A Critic for Language Model Generationcited by 97

50 most cited of 78 All 78 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 9; 2026: 6.

15 papers.

Sort:

By area:

  1. 2025Generative RLHF-V: Learning Principles from Multi-modal Human Preferencecited by 13
  2. 2025Pluralistic Off-policy Evaluation and Alignmentcited by 11
  3. 2025Routine: A Structural Planning Framework for LLM Agent System in Enterprisecited by 9
  4. 2025Unlocking the Essence of Beauty: Advanced Aesthetic Reasoning with Relative-Absolute Policy Optimizationcited by 7
  5. 2025Eliciting and Analyzing Emergent Misalignment in State-of-the-Art Large Language Modelscited by 4
  6. 2025Multidimensional Rubric-oriented Reward Model Learning via Geometric Projection Reference Constraintscited by 4
  7. 2025We Think, Therefore We Align LLMs to Helpful, Harmless and Honest Before They Go Wrongcited by 3
  8. 2025On the Role of Preference Variance in Preference Optimizationcited by 3
  9. 2026Safety, Security, and Cognitive Risks in World Modelscited by 3
  10. 2026A Survey on Rubric-Guided Reinforcement Learning for Language Modelscited by 3
  11. 2025GTAlign: Game-Theoretic Alignment of LLM Assistants for Social Welfarecited by 2
  12. 2026Overton Pluralistic Reinforcement Learning for Large Language Modelscited by 1
3 more
  1. 2026Reward-Zero: Language Embedding Driven Implicit Reward Mechanisms for Reinforcement Learningcited by 0
  2. 2026ClaHF: A Human Feedback-inspired Reinforcement Learning Framework for Improving Classification Taskscited by 0
  3. 2026The Missing Coefficients: Bayesian Pairwise Merging for Model Personalizationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Elephant in the Room: Unveiling the Impact of Reward Model Quality in Alignment
  2. 2024RMB: Comprehensively Benchmarking Reward Models in LLM Alignment
  3. 2026Reward Modeling for Reinforcement Learning-Based LLM Reasoning: Design, Challenges, and Evaluation
  4. 2025Reward-aware Preference Optimization: A Unified Mathematical Framework for Model Alignment
  5. 2025Beyond Monolithic Rewards: A Hybrid and Multi-Aspect Reward Optimization for MLLM Alignment
  6. 2024Reward-Augmented Data Enhances Direct Preference Alignment of LLMs
  7. 2026The End of Reward Engineering: How LLMs Are Redefining Multi-Agent Coordination
  8. 2025Reusing Embeddings: Reproducible Reward Model Research in Large Language Model Alignment without GPUs