mlsanityData to 14 Oct 2026

Sailing by the Stars: A Survey on Reward Models and Learning Strategies for Learning from Rewards

arXiv 2505.02686 ↗First posted 5 May 2025cs.CLcited by 14

On the map: LLM reasoning → Reinforcement learning for reasoning

also in 1 more
Abstract (arXiv)

Recent developments in Large Language Models (LLMs) have shifted from pre-training scaling to post-training and test-time scaling. Across these developments, a key unified paradigm has arisen: Learning from Rewards, where reward signals act as the guiding stars to steer LLM behavior. It has underpinned a wide range of prevalent techniques, such as reinforcement learning (RLHF, RLAIF, DPO, and GRPO), reward-guided decoding, and post-hoc correction. Crucially, this paradigm enables the transition from passive learning from static data to active learning from dynamic feedback. This endows LLMs with aligned preferences and deep reasoning capabilities for diverse tasks. In this survey, we present a comprehensive overview of learning from rewards, from the perspective of reward models and learning strategies across training, inference, and post-inference stages. We further discuss the benchmarks for reward models and the primary applications. Finally we highlight the challenges and future directions. We maintain a paper collection at https://github.com/bobxwu/learning-from-rewards-llm-papers.

Key facts. Cited by 14 papers in the map (data through 14 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 196 of its 249 references that are in the map. First posted on arXiv 5 May 2025.

Builds on

196 of 249 references.

Closest ancestors · 36

Same direction, newest first

  1. 2025Process Reward Models That Think23 Apr
  2. 2025Not All Rollouts are Useful: Down-Sampling Rollouts in LLM Reinforcement Learning18 Apr
  3. 2025Improving Generalization in Intent Detection: GRPO with Reward-Based Curriculum Sampling18 Apr
  4. 2025Efficient Process Reward Model Training via Active Learning14 Apr
  5. 2025Right Question is Already Half the Answer: Fully Unsupervised LLM Reasoning Incentivization8 Apr
  6. 2025Synthetic Data Generation & Multi-Step RL for Reasoning & Tool Use7 Apr
  7. 2025Inference-Time Scaling for Generalist Reward Modeling3 Apr
  8. 2025GenPRM: Scaling Test-Time Compute of Process Reward Models via Generative Reasoning1 Apr
  9. 2025CrowdVLM-R1: Expanding R1 Ability to Vision Language Model for Crowd Counting using Fuzzy Group Relative Policy Reward31 Mar
  10. 2025R-PRM: Reasoning-Driven Process Reward Modeling27 Mar
  11. 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scale18 Mar
  12. 2025R1-VL: Learning to Reason with Multimodal Large Language Models via Step-wise Group Relative Policy Optimization17 Mar
24 more
  1. 2025Process-based Self-Rewarding Language Models5 Mar
  2. 2025Agentic Reward Modeling: Integrating Human Preferences with Verifiable Correctness Signals for Reliable Reward Systems26 Feb
  3. 2025SWE-RL: Advancing LLM Reasoning via Reinforcement Learning on Open Software Evolution25 Feb
  4. 2025Process Reward Models for LLM Agents: Practical Framework and Directions14 Feb
  5. 2025Exploring the Limit of Outcome Reward for Learning Mathematical Reasoning10 Feb
  6. 2025Process Reinforcement through Implicit Rewards3 Feb
  7. 2025The Lessons of Developing Process Reward Models in Mathematical Reasoning13 Jan
  8. 2025PRMBench: A Fine-grained and Challenging Benchmark for Process-Level Reward Models6 Jan
  9. 2024ProcessBench: Identifying Process Errors in Mathematical Reasoning9 Dec
  10. 2024Free Process Rewards without Process Labels2 Dec
  11. 2024Self-Generated Critiques Boost Reward Modeling for Language Models25 Nov
  12. 2024Skywork-Reward: Bag of Tricks for Reward Modeling in LLMs24 Oct
  13. 2024Process Reward Model with Q-Value Rankings15 Oct
  14. 2024Generative Verifiers: Reward Modeling as Next-Token Prediction27 Aug
  15. 2024ReST-MCTS*: LLM Self-Training via Process Reward Guided Tree Search6 Jun
  16. 2024Improve Mathematical Reasoning in Language Models by Automated Process Supervision5 Jun
  17. 2024Learning Planning-based Reasoning by Trajectories Collection and Process Reward Synthesizing1 Feb
  18. 2023Math-Shepherd: Verify and Reinforce LLMs Step-by-step without Human Annotations14 Dec
  19. 2023Let's Verify Step by Step31 May
  20. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Model29 May
  21. 2023RAFT: Reward rAnked FineTuning for Generative Foundation Model Alignment13 Apr
  22. 2022Constitutional AI: Harmlessness from AI Feedback15 Dec
  23. 2022Solving math word problems with process- and outcome-based feedback25 Nov
  24. 2017Proximal Policy Optimization Algorithms20 Jul

Other references · 160

  1. 2022Training language models to follow instructions with human feedbackcited by 15,971
  2. 2024The Llama 3 Herd of Modelscited by 14,715
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  4. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  5. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  6. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  7. 2020Scaling Laws for Neural Language Modelscited by 6,586
  8. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,708
  9. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,371
  10. 2023Reflexion: Language Agents with Verbal Reinforcement Learningcited by 4,276
  11. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,720
  12. 2023Self-Refine: Iterative Refinement with Self-Feedbackcited by 3,442
38 more
  1. 2023A Survey on Large Language Model based Autonomous Agentscited by 2,242
  2. 2019Fine-Tuning Language Models from Human Preferencescited by 2,150
  3. 2023Improving Factuality and Reasoning in Language Models through Multiagent Debatecited by 1,784
  4. 2021WebGPT: Browser-assisted question-answering with human feedbackcited by 1,745
  5. 2016Concrete Problems in AI Safetycited by 1,642
  6. 2025Search-R1: Training LLMs to Reason and Leverage Search Engines with Reinforcement Learningcited by 1,444
  7. 2024OlympiadBench: A Challenging Benchmark for Promoting AGI with Olympiad-Level Bilingual Multimodal Scientific Problemscited by 1,136
  8. 2023ImageReward: Learning and Evaluating Human Preferences for Text-to-Image Generationcited by 1,045
  9. 2023Large Language Models Cannot Self-Correct Reasoning Yetcited by 1,017
  10. 2023Encouraging Divergent Thinking in Large Language Models through Multi-Agent Debatecited by 1,011
  11. 2024Large Language Monkeys: Scaling Inference Compute with Repeated Samplingcited by 991
  12. 2023Teaching Large Language Models to Self-Debugcited by 988
  13. 2024SimPO: Simple Preference Optimization with a Reference-Free Rewardcited by 963
  14. 2023Reasoning with Language Model is Planning with World Modelcited by 861
  15. 2023BeaverTails: Towards Improved Safety Alignment of LLM via a Human-Preference Datasetcited by 860
  16. 2023Diffusion Model Alignment Using Direct Preference Optimizationcited by 790
  17. 2023CRITIC: Large Language Models Can Self-Correct with Tool-Interactive Critiquingcited by 783
  18. 2023Aligning Large Multimodal Models with Factually Augmented RLHFcited by 712
  19. 2023Safe RLHF: Safe Reinforcement Learning from Human Feedbackcited by 656
  20. 2023WizardMath: Empowering Mathematical Reasoning for Large Language Models via Reinforced Evol-Instructcited by 625
  21. 2023RLAIF vs. RLHF: Scaling Reinforcement Learning from Human Feedback with AI Feedbackcited by 625
  22. 2022Improving alignment of dialogue agents via targeted human judgementscited by 531
  23. 2025Visual-RFT: Visual Reinforcement Fine-Tuningcited by 513
  24. 2023Reinforced Self-Training (ReST) for Language Modelingcited by 500
  25. 2025Video-R1: Reinforcing Video Reasoning in MLLMscited by 484
  26. 2023RRHF: Rank Responses to Align Language Models with Human Feedback without tearscited by 463
  27. 2024Training Language Models to Self-Correct via Reinforcement Learningcited by 426
  28. 2023Fine-Grained Human Feedback Gives Better Rewards for Language Model Trainingcited by 401
  29. 2025ReTool: Reinforcement Learning for Strategic Tool Use in LLMscited by 392
  30. 2023Logic-LM: Empowering Large Language Models with Symbolic Solvers for Faithful Logical Reasoningcited by 391
  31. 2025RAGEN: Understanding Self-Evolution in LLM Agents via Multi-Turn Reinforcement Learningcited by 386
  32. 2022Self-critiquing models for assisting human evaluatorscited by 380
  33. 2024Interpretable Preferences via Multi-Objective Reward Modeling and Mixture-of-Expertscited by 376
  34. 2023Language Models can Solve Computer Taskscited by 372
  35. 2024MLLM-as-a-Judge: Assessing Multimodal LLM-as-a-Judge with Vision-Language Benchmarkcited by 372
  36. 2025ToolRL: Reward is All Tool Learning Needscited by 362
  37. 2023Toxicity in ChatGPT: Analyzing Persona-assigned Language Modelscited by 360
  38. 2023Check Your Facts and Try Again: Improving Large Language Models with External Knowledge and Automated Feedbackcited by 353

50 most cited of 160 All 160 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 8; 2026: 6.

14 papers.

Sort:

By area:

  1. 2025A Survey of Reinforcement Learning for Large Reasoning Modelscited by 180
  2. 2025Towards Storage-Efficient Visual Document Retrieval: An Empirical Study on Reducing Patch-Level Embeddingscited by 26
  3. 2025AgentMath: Empowering Mathematical Reasoning for Large Language Models via Tool-Augmented Agentcited by 17
  4. 2025Training LLMs for EHR-Based Reasoning Tasks via Reinforcement Learningcited by 14
  5. 2025Enhancing Large Language Model Reasoning with Reward Models: An Analytical Surveycited by 8
  6. 2025A Survey of Inductive Reasoning for Large Language Modelscited by 8
  7. 2026Reward Modeling for Reinforcement Learning-Based LLM Reasoning: Design, Challenges, and Evaluationcited by 7
  8. 2025Unsupervised Hallucination Detection by Inspecting Reasoning Processescited by 5
  9. 2025Multidimensional Rubric-oriented Reward Model Learning via Geometric Projection Reference Constraintscited by 4
  10. 2026VERTIGO: Visual Preference Optimization for Cinematic Camera Trajectory Generationcited by 3
  11. 2026Can Thinking Models Think to Detect Hateful Memes?cited by 2
  12. 2026Modularized Reinforcement Learning on LLMs: From MDP Creation to Exploration and Learningcited by 2
2 more
  1. 2026Adapting Reinforcement Learning with Chain-of-Thought Supervision for Explainable Detection of Hateful and Propagandistic Memescited by 1
  2. 2026Unsupervised Post-Training of Foundation Models: A Surveycited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025GRAM: A Generative Foundation Reward Model for Reward Generalization
  2. 2024RewardBench: Evaluating Reward Models for Language Modeling
  3. 2025Generalist Reward Models: Found Inside Large Language Models
  4. 2025Better Language Model-Based Judging Reward Modeling through Scaling Comprehension Boundaries
  5. 2025Reusing Embeddings: Reproducible Reward Model Research in Large Language Model Alignment without GPUs
  6. 2026Small Reward Models via Backward Inference
  7. 2026Likelihood-Based Reward Designs for General LLM Reasoning
  8. 2025A Survey on Progress in LLM Alignment from the Perspective of Reward Design