mlsanityData to 14 Oct 2026

Libra: Assessing and Improving Reward Model by Learning to Think

arXiv 2507.21645 ↗First posted 29 Jul 2025cs.CLcited by 4

On the map: LLM reasoning → Reinforcement learning for reasoning

also in 1 more
Abstract (arXiv)

Reinforcement learning (RL) has significantly improved the reasoning ability of large language models. However, current reward models underperform in challenging reasoning scenarios and predominant RL training paradigms rely on rule-based or reference-based rewards, which impose two critical limitations: 1) the dependence on finely annotated reference answer to attain rewards; and 2) the requirement for constrained output format. These limitations fundamentally hinder further RL data scaling and sustained enhancement of model reasoning performance. To address these limitations, we propose a comprehensive framework for evaluating and improving the performance of reward models in complex reasoning scenarios. We first present a reasoning-oriented benchmark (Libra Bench), systematically constructed from a diverse collection of challenging mathematical problems and advanced reasoning models, to address the limitations of existing reward model benchmarks in reasoning scenarios. We further introduce a novel approach for improving the generative reward model via learning-to-think methodologies. Based on the proposed approach, we develop Libra-RM series, a collection of generative reward models with reasoning capabilities that achieve state-of-the-art results on various benchmarks. Comprehensive downstream experiments are conducted and the experimental results demonstrate the correlation between our Libra Bench and downstream application, and the potential of Libra-RM to further improve reasoning models with unlabeled data.

Key facts. Cited by 4 papers in the map (data through 14 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 54 of its 69 references that are in the map. First posted on arXiv 29 Jul 2025.

Builds on

54 of 69 references.

Closest ancestors · 15

Same direction, newest first

  1. 2025Reward Reasoning Model20 May
  2. 2025RM-R1: Reward Modeling as Reasoning5 May
  3. 2025VAPO: Efficient and Reliable Reinforcement Learning for Advanced Reasoning Tasks7 Apr
  4. 2025Inference-Time Scaling for Generalist Reward Modeling3 Apr
  5. 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scale18 Mar
  6. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning22 Jan
  7. 2025Kimi k1.5: Scaling Reinforcement Learning with LLMs22 Jan
  8. 2025PRMBench: A Fine-grained and Challenging Benchmark for Process-Level Reward Models6 Jan
  9. 2024ProcessBench: Identifying Process Errors in Mathematical Reasoning9 Dec
  10. 2024Tulu 3: Pushing Frontiers in Open Language Model Post-Training22 Nov
  11. 2024Skywork-Reward: Bag of Tricks for Reward Modeling in LLMs24 Oct
  12. 2024Generative Verifiers: Reward Modeling as Next-Token Prediction27 Aug
3 more
  1. 2024RLHF Workflow: From Reward Modeling to Online RLHF13 May
  2. 2024KTO: Model Alignment as Prospect Theoretic Optimization2 Feb
  3. 2017Proximal Policy Optimization Algorithms20 Jul

Other references · 39

  1. 2024The Llama 3 Herd of Modelscited by 14,715
  2. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  3. 2025Qwen3 Technical Reportcited by 8,820
  4. 2024GPT-4o System Cardcited by 4,588
  5. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,720
  6. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,224
  7. 2024A Survey on LLM-as-a-Judgecited by 1,423
  8. 2024Qwen2.5-Math Technical Report: Toward Mathematical Expert Model via Self-Improvementcited by 1,090
  9. 2024Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluatorscited by 897
  10. 2024DeepSeek-V3 Technical Reportcited by 811
  11. 2024From Crowdsourced Data to High-Quality Benchmarks: Arena-Hard and BenchBuilder Pipelinecited by 500
  12. 2024RewardBench: Evaluating Reward Models for Language Modelingcited by 499
27 more
  1. 2024Interpretable Preferences via Multi-Objective Reward Modeling and Mixture-of-Expertscited by 376
  2. 2023JudgeLM: Fine-tuned Large Language Models are Scalable Judgescited by 367
  3. 2023Iterative Preference Learning from Human Feedback: Bridging Theory and Practice for RLHF under KL-Constraintcited by 350
  4. 2024JudgeBench: A Benchmark for Evaluating LLM-based Judgescited by 318
  5. 2024HelpSteer2: Open-source dataset for training top-performing reward modelscited by 236
  6. 2023Secrets of RLHF in Large Language Models Part I: PPOcited by 229
  7. 2024RM-Bench: Benchmarking Reward Models of Language Models with Subtlety and Stylecited by 176
  8. 2025Light-R1: Curriculum SFT, DPO and RL for Long COT from Scratch and Beyondcited by 175
  9. 2025Skywork Open Reasoner 1 Technical Reportcited by 165
  10. 2025Seed1.5-Thinking: Advancing Superb Reasoning Models with Reinforcement Learningcited by 161
  11. 2025RewardBench 2: Advancing Reward Model Evaluationcited by 150
  12. 2024Nemotron-4 340B Technical Reportcited by 144
  13. 2025AgentRewardBench: Evaluating Automatic Evaluations of Web Agent Trajectoriescited by 108
  14. 2024Critique-out-Loud Reward Modelscited by 106
  15. 2024How to Evaluate Reward Models for RLHFcited by 88
  16. 2025Learning to Plan & Reason for Evaluation with Thinking-LLM-as-a-Judgecited by 78
  17. 2025JudgeLRM: Large Reasoning Models as a Judgecited by 78
  18. 2025J1: Incentivizing Thinking in LLM-as-a-Judge via Reinforcement Learningcited by 75
  19. 2024Self-play with Execution Feedback: Improving Instruction-following Capabilities of Large Language Modelscited by 73
  20. 2024Self-Taught Evaluatorscited by 72
  21. 2024MJ-Bench: Is Your Multimodal Reward Model Really a Good Judge for Text-to-Image Generation?cited by 66
  22. 2024RMB: Comprehensively Benchmarking Reward Models in LLM Alignmentcited by 61
  23. 2024M-RewardBench: Evaluating Reward Models in Multilingual Settingscited by 50
  24. 2025Multimodal RewardBench: Holistic Evaluation of Reward Models for Vision Language Modelscited by 44
  25. 2025reWordBench: Benchmarking and Improving the Robustness of Reward Models with Transformed Inputscited by 25
  26. 2024RAG-RewardBench: Benchmarking Reward Models in Retrieval Augmented Generation for Preference Alignmentcited by 23
  27. 2025VLRMBench: A Comprehensive and Challenging Benchmark for Vision-Language Reward Modelscited by 19

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 3; 2026: 1.

4 papers.

  1. 2025A Survey of Reinforcement Learning for Large Reasoning Modelscited by 180
  2. 2025Introducing LongCat-Flash-Thinking: A Technical Reportcited by 22
  3. 2025Enhancing Large Language Model Reasoning with Reward Models: An Analytical Surveycited by 8
  4. 2026On the Generalization Gap in Self-Evolving Language Model Reasoningcited by 4

Similar, not cited

Close in wording, no citation either way

  1. 2025Think-RM: Enabling Long-Horizon Reasoning in Generative Reward Models
  2. 2025Beyond Correctness: Confidence-Aware Reward Modeling for Enhancing Large Language Model Reasoning
  3. 2026Reward Modeling for Reinforcement Learning-Based LLM Reasoning: Design, Challenges, and Evaluation
  4. 2026Reason Only When Needed: Efficient Generative Reward Modeling via Model-Internal Uncertainty
  5. 2026Likelihood-Based Reward Designs for General LLM Reasoning
  6. 2025Crossing the Reward Bridge: Expanding RL with Verifiable Rewards Across Diverse Domains
  7. 2025Rethinking Reasoning Quality in Large Language Models through Enhanced Chain-of-Thought via RL
  8. 2025VerifyBench: Benchmarking Reference-based Reward Systems for Large Language Models