mlsanityData to 14 Oct 2026

Generalist Reward Models: Found Inside Large Language Models

arXiv 2506.23235 ↗First posted 29 Jun 2025cs.CLcited by 34

On the map: LLM alignment / RLHF / preference optimisation → Reward modeling and RLHF

Abstract (arXiv)

The alignment of Large Language Models (LLMs) is critically dependent on reward models trained on costly human preference data. While recent work explores bypassing this cost with AI feedback, these methods often lack a rigorous theoretical foundation. In this paper, we discover that a powerful generalist reward model is already latently present within any LLM trained via standard next-token prediction. We prove that this endogenous reward is not a heuristic, but is theoretically equivalent to a reward function learned through offline inverse reinforcement learning. This connection allows us to directly elicit a high-quality reward signal from a base (pre-trained or supervised fine-tuned) model without any further training. Critically, we also prove that subsequent reinforcement learning using this endogenous reward leads to a policy with a provably superior error bound compared to the base model. To our best knowledge, this is the first theoretical proof of the effectiveness of reinforcement learning for LLMs. Our experiments validate this theory, demonstrating that our method not only outperforms existing LLM-as-a-judge approaches but can also surpass explicitly trained reward models. These findings suggest that the reward modeling stage can be replaced by a principled method of eliciting the knowledge already captured during pre-training, heralding a more efficient, powerful, and scalable paradigm for LLMs alignment as well as multi-modal models.

Key facts. Cited by 34 papers in the map (data through 14 Oct 2026). Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 37 of its 50 references that are in the map. First posted on arXiv 29 Jun 2025.

Builds on

37 of 50 references.

Closest ancestors · 9

Same direction, newest first

  1. 2025RewardAnything: Generalizable Principle-Following Reward Models4 Jun
  2. 2025RewardBench 2: Advancing Reward Model Evaluation2 Jun
  3. 2024RM-Bench: Benchmarking Reward Models of Language Models with Subtlety and Style21 Oct
  4. 2024Generative Reward Models2 Oct
  5. 2023Everyone Deserves A Reward: Learning Customized Human Preferences6 Sep
  6. 2023RLAIF vs. RLHF: Scaling Reinforcement Learning from Human Feedback with AI Feedback1 Sep
  7. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Model29 May
  8. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback12 Apr
  9. 2017Deep reinforcement learning from human preferences12 Jun

Other references · 28

  1. 2017Attention Is All You Needcited by 60,361
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2022Training language models to follow instructions with human feedbackcited by 15,971
  4. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  5. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  6. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  7. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,371
  8. 2024Qwen2.5 Technical Reportcited by 4,942
  9. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 3,050
  10. 2024A Survey on LLM-as-a-Judgecited by 1,423
  11. 2022Solving Quantitative Reasoning Problems with Language Modelscited by 1,392
  12. 2024OlympiadBench: A Challenging Benchmark for Promoting AGI with Olympiad-Level Bilingual Multimodal Scientific Problemscited by 1,136
16 more
  1. 2023A General Theoretical Paradigm to Understand Learning from Human Preferencescited by 912
  2. 2017Reinforcement Learning with Deep Energy-Based Policiescited by 669
  3. 2024Generative Verifiers: Reward Modeling as Next-Token Predictioncited by 480
  4. 2018An Algorithmic Perspective on Imitation Learningcited by 459
  5. 2019A Theory of Regularized Markov Decision Processescited by 286
  6. 2022Is Reinforcement Learning (Not) for Natural Language Processing: Benchmarks, Baselines, and Building Blocks for Natural Language Policy Optimizationcited by 246
  7. 2025Learning to Reason without External Rewardscited by 210
  8. 2021IQ-Learn: Inverse soft-Q Learning for Imitationcited by 193
  9. 2020Toward the Fundamental Limits of Imitation Learningcited by 96
  10. 2020Error Bounds of Imitating Policies and Environmentscited by 90
  11. 2024Aligning to Thousands of Preferences via System Message Generalizationcited by 84
  12. 2023Language Model Self-improvement by Reinforcement Learning Contemplationcited by 75
  13. 2025Reinforcement Pre-Trainingcited by 40
  14. 2023Query-Dependent Prompt Evaluation and Optimization with Offline Inverse RLcited by 37
  15. 2024Imitating Language via Scalable Inverse Reinforcement Learningcited by 21
  16. 2022Understanding Adversarial Imitation Learning in Small Sample Regime: A Stage-coupled Analysiscited by 13

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 11; 2026: 23.

34 papers.

Sort:

By area:

  1. 2026Reinforcement Learning via Self-Distillationcited by 332
  2. 2025A Survey of Self-Evolving Agents: What, When, How, and Where to Evolve on the Path to Artificial Super Intelligencecited by 170
  3. 2025Uniworld-V2: Reinforce Image Editing with Diffusion Negative-aware Finetuning and MLLM Implicit Feedbackcited by 79
  4. 2025GTM: Simulating the World of Tools for AI Agentscited by 14
  5. 2025LegalDelta: Enhancing Legal Reasoning in LLMs via Reinforcement Learning with Chain-of-Thought Guided Information Gaincited by 12
  6. 2025Random Policy Valuation is Enough for LLM Reasoning with Verifiable Rewardscited by 10
  7. 2026LongR: Unleashing Long-Context Reasoning via Reinforcement Learning with Dense Utility Rewardscited by 10
  8. 2025Semantic Voting: A Self-Evaluation-Free Approach for Efficient LLM Self-Improvement on Unverifiable Open-ended Taskscited by 7
  9. 2026V0: A Generalist Value Model for Any Policy at State Zerocited by 7
  10. 2026Reward Modeling for Reinforcement Learning-Based LLM Reasoning: Design, Challenges, and Evaluationcited by 7
  11. 2025PVPO: Pre-Estimated Value-Based Policy Optimization for Agentic Reasoningcited by 4
  12. 2026CAST: Game Solvers as Turn-Level Teachers for LLM Agentscited by 3
22 more
  1. 2025SSPO: Self-traced Step-wise Preference Optimization for Process Supervision and Reasoning Compressioncited by 2
  2. 2026Quark Medical Alignment: A Holistic Multi-Dimensional Alignment and Collaborative Optimization Paradigmcited by 2
  3. 2026Believe Your Model: Distribution-Guided Confidence Calibrationcited by 2
  4. 2025Beyond Imitation: Recovering Dense Rewards from Demonstrationscited by 1
  5. 2025Beyond Token-level Supervision: Unlocking the Potential of Decoding-based Regression via Reinforcement Learningcited by 1
  6. 2026AERO: Autonomous Evolutionary Reasoning Optimization via Endogenous Dual-Loop Feedbackcited by 1
  7. 2026Stabilizing Policy Optimization via Logits Convexitycited by 1
  8. 2026Beyond Length Scaling: Synergizing Breadth and Depth for Generative Reward Modelscited by 1
  9. 2026Unleashing Implicit Rewards: Prefix-Value Learning for Distribution-Level Optimizationcited by 1
  10. 2026Alignment Tuning for Large Language Models: A Data-Centric Lens on Alignment Data Pipelinescited by 1
  11. 2025RLFR: Extending Reinforcement Learning for LLMs with Flow Environmentcited by 0
  12. 2026PaperGuide: Making Small Language-Model Paper-Reading Agents More Efficientcited by 0
  13. 2026IRIS: Implicit Reward-Guided Internal Sifting for Mitigating Multimodal Hallucinationcited by 0
  14. 2026RL-RIG: A Generative Spatial Reasoner via Intrinsic Reflectioncited by 0
  15. 2026From the Inside Out: Progressive Distribution Refinement for Confidence Calibrationcited by 0
  16. 2026Off-Policy Value-Based Reinforcement Learning for Large Language Modelscited by 0
  17. 2026LC-ERD: Mining Latent Logic for Self-Evolving Reasoning via Consistency-Regulated Reward Decompositioncited by 0
  18. 2026Directional Alignment Mitigates Reward Hacking in Reinforcement Learning for Language Modelscited by 0
  19. 2026Momentum for Reasoning: Dense Intrinsic Signals in Policy Optimizationcited by 0
  20. 2026Provably Efficient Policy-Reward Co-Pretraining for Adversarial Imitation Learningcited by 0
  21. 2026REAR: Test-time Preference Realignment through Reward Decompositioncited by 0
  22. 2026Rollout Efficiency in Reinforcement Learning for Reasoning Large Language Models: A Taxonomy and Future Directionscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Inverse Reinforcement Learning Meets Large Language Model Post-Training: Basics, Advances, and Opportunities
  2. 2025Response-Level Rewards Are All You Need for Online Reinforcement Learning in LLMs: A Mathematical Perspective
  3. 2024Insights from the Inverse: Reconstructing LLM Training Goals Through Inverse Reinforcement Learning
  4. 2024On the Modeling Capabilities of Large Language Models for Sequential Decision Making
  5. 2024Regularizing Hidden States Enables Learning Generalizable Reward Model for LLMs
  6. 2025Sailing by the Stars: A Survey on Reward Models and Learning Strategies for Learning from Rewards
  7. 2025GRAM: A Generative Foundation Reward Model for Reward Generalization
  8. 2025Reusing Embeddings: Reproducible Reward Model Research in Large Language Model Alignment without GPUs