mlsanityData to 14 Oct 2026

Hybrid Reinforcement: When Reward Is Sparse, It's Better to Be Dense

arXiv 2510.07242 ↗First posted 8 Oct 2025cs.CL cs.LGcited by 20

On the map: LLM reasoning → Reinforcement learning for reasoning

also in 1 more
Abstract (arXiv)

Post-training for reasoning of large language models (LLMs) increasingly relies on verifiable rewards: deterministic checkers that provide 0-1 correctness signals. While reliable, such binary feedback is brittle--many tasks admit partially correct or alternative answers that verifiers under-credit, and the resulting all-or-nothing supervision limits learning. Reward models offer richer, continuous feedback, which can serve as a complementary supervisory signal to verifiers. We introduce HERO (Hybrid Ensemble Reward Optimization), a reinforcement learning framework that integrates verifier signals with reward-model scores in a structured way. HERO employs stratified normalization to bound reward-model scores within verifier-defined groups, preserving correctness while refining quality distinctions, and variance-aware weighting to emphasize challenging prompts where dense signals matter most. Across diverse mathematical reasoning benchmarks, HERO consistently outperforms RM-only and verifier-only baselines, with strong gains on both verifiable and hard-to-verify tasks. Our results show that hybrid reward design retains the stability of verifiers while leveraging the nuance of reward models to advance reasoning.

Key facts. Cited by 20 papers in the map (data through 14 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 37 of its 42 references that are in the map. First posted on arXiv 8 Oct 2025.

Builds on

37 of 42 references.

Closest ancestors · 16

Same direction, newest first

  1. 2025A Survey of Reinforcement Learning for Large Reasoning Models10 Sep
  2. 2025VerifyBench: A Systematic Benchmark for Evaluating Reasoning Verifiers Across Domains14 Jul
  3. 2025Skywork-Reward-V2: Scaling Preference Data Curation via Human-AI Synergy2 Jul
  4. 2025OctoThinker: Mid-training Incentivizes Reinforcement Learning Scaling25 Jun
  5. 2025Reinforcing General Reasoning without Verifiers27 May
  6. 2025General-Reasoner: Advancing LLM Reasoning Across All Domains20 May
  7. 2025TinyV: Reducing False Negatives in Verification Improves RL for LLM Reasoning20 May
  8. 2025A Minimalist Approach to LLM Reasoning: from Rejection Sampling to Reinforce15 Apr
  9. 2025Crossing the Reward Bridge: Expanding RL with Verifiable Rewards Across Diverse Domains31 Mar
  10. 2025SimpleRL-Zoo: Investigating and Taming Zero Reinforcement Learning for Open Base Models in the Wild24 Mar
  11. 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scale18 Mar
  12. 2025Agentic Reward Modeling: Integrating Human Preferences with Verifiable Correctness Signals for Reliable Reward Systems26 Feb
4 more
  1. 2025Big-Math: A Large-Scale, High-Quality Math Dataset for Reinforcement Learning in Language Models24 Feb
  2. 2025Exploring the Limit of Outcome Reward for Learning Mathematical Reasoning10 Feb
  3. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning22 Jan
  4. 2025The Lessons of Developing Process Reward Models in Mathematical Reasoning13 Jan

Other references · 21

  1. 2024The Llama 3 Herd of Modelscited by 14,715
  2. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  3. 2025Qwen3 Technical Reportcited by 8,820
  4. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  5. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,371
  6. 2024HybridFlow: A Flexible and Efficient RLHF Frameworkcited by 2,057
  7. 2022Solving Quantitative Reasoning Problems with Language Modelscited by 1,392
  8. 2024OlympiadBench: A Challenging Benchmark for Promoting AGI with Olympiad-Level Bilingual Multimodal Scientific Problemscited by 1,136
  9. 2024Qwen2.5-Math Technical Report: Toward Mathematical Expert Model via Self-Improvementcited by 1,090
  10. 2025Humanity's Last Examcited by 491
  11. 2025Towards Reasoning Era: A Survey of Long Chain-of-Thought for Reasoning Large Language Modelscited by 384
  12. 2025AIMO-2 Winning Solution: Building State-of-the-Art Mathematical Reasoning Models with OpenMathReasoning datasetcited by 174
9 more
  1. 2025Reinforcement Learning with Rubric Anchorscited by 113
  2. 2018Leveraging Grammar and Reinforcement Learning for Neural Program Synthesiscited by 84
  3. 2024AceMath: Advancing Frontier Math Reasoning with Post-Training and Reward Modelingcited by 77
  4. 2025MegaScience: Pushing the Frontiers of Post-Training Datasets for Science Reasoningcited by 59
  5. 2025xVerify: Efficient Answer Verifier for Reasoning Model Evaluationscited by 55
  6. 2025Learning to Reason for Long-Form Story Generationcited by 34
  7. 2025RealMath: A Continuous Benchmark for Evaluating Language Models on Research-Level Mathematicscited by 31
  8. 2025Beyond Verifiable Rewards: Scaling Reinforcement Learning for Language Models to Unverifiable Datacited by 29
  9. 2025Lessons from Training Grounded LLMs with Verifiable Rewardscited by 8

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 2; 2026: 18.

20 papers.

Sort:

By area:

  1. 2025CodeV: Code with Images for Faithful Visual Reasoning via Tool-Aware Policy Optimizationcited by 26
  2. 2026RAGEN-2: Reasoning Collapse in Agentic RLcited by 20
  3. 2026PROGRESSLM: Towards Progress Reasoning in Vision-Language Modelscited by 15
  4. 2025Certified Self-Consistency: Statistical Guarantees and Test-Time Training for Reliable Reasoning in LLMscited by 12
  5. 2026Silence the Judge: Reinforcement Learning with Self-Verifier via Latent Geometric Clusteringcited by 7
  6. 2026AVSD: Adaptive-View Self-Distillation by Balancing Consensus and Teacher-Specific Privileged Signalscited by 7
  7. 2026SA-VLA: Spatially-Aware Flow-Matching for Vision-Language-Action Reinforcement Learningcited by 3
  8. 2026Reasoning over mathematical objects: on-policy reward modeling and test time aggregationcited by 3
  9. 2026ExecCritic: Learn to Test, Test to Improve for Coding Agentscited by 3
  10. 2026RLVP: Penalize the Path, Reward the Outcomecited by 2
  11. 2026When Errors Can Be Beneficial: A Categorization of Imperfect Rewards for Policy Gradientcited by 1
  12. 2026Coverage, Not Targeting: A Structural Regime in Multi-Turn Agent Credit Assignmentcited by 1
8 more
  1. 2026Miner:Mining Intrinsic Mastery for Data-Efficient RL in Large Reasoning Modelscited by 0
  2. 2026Configurable Reward Model for Balanced Safety Alignmentcited by 0
  3. 2026Internalizing Geometric Law: Learning from Solver Residuals for Precision-Critical Generationcited by 0
  4. 2026ProcessThinker: Enhancing Multi-modal Large Language Models Reasoning via Rollout-based Process Rewardcited by 0
  5. 2026Enhancing VLM Reward Models Through Structure-Aware Fine-Tuningcited by 0
  6. 2026ARC: Fair Relative Advantage Comparison in Open-Ended Real-World Interactioncited by 0
  7. 2026RLVR landscapes for iterated multiplications can be benign: Insights from spin-glass theorycited by 0
  8. 2026Beyond Binary Verifiable Rewards: Behavioral Self-Rewarding for SWE Agentscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025The Good, The Bad, and The Hybrid: A Reward Structure Showdown in Reasoning Models Training
  2. 2026Likelihood-Based Reward Designs for General LLM Reasoning
  3. 2025ConfClip: Confidence-Weighted and Clipped Reward for Reinforcement Learning in LLMs
  4. 2026Multilingual Verifier Bias in RLVR: Benchmark, Rollout Diagnosis, and the Cross-Lingual Selection Bottleneck
  5. 2025A Simple "Motivation" Can Enhance Reinforcement Finetuning of Large Reasoning Models
  6. 2025Beyond Monolithic Rewards: A Hybrid and Multi-Aspect Reward Optimization for MLLM Alignment
  7. 2025The Climb Carves Wisdom Deeper Than the Summit: On the Noisy Rewards in Learning to Reason
  8. 2026Gradients Know What Outcomes Don't: Unlocking Reinforcement Learning for LLM Reasoning with Gradient-Aligned Rewards