# Hybrid Reinforcement: When Reward Is Sparse, It's Better to Be Dense arXiv: https://arxiv.org/abs/2510.07242 ยท HTML: https://mlsanity.com/paper/2510.07242/ ## Key facts - Cited by 20 papers in the map (data through 14 Oct 2026). - Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. - Builds on 37 of its 42 references that are in the map. - First posted on arXiv 8 Oct 2025. ## Placed in - LLM reasoning / [Reinforcement learning for reasoning](https://mlsanity.com/lr/reinforcement-learning-for-reasoning/) - LLM alignment / RLHF / preference optimisation / [Process reward models for reasoning](https://mlsanity.com/alignment/process-reward-models-for-reasoning/) ## Builds on (most cited first, 30 of 37) - [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024) - [DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models](https://mlsanity.com/paper/2402.03300/) (2024) - [Qwen3 Technical Report](https://mlsanity.com/paper/2505.09388/) (2025) - [Evaluating Large Language Models Trained on Code](https://mlsanity.com/paper/2107.03374/) (2021) - [DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning](https://mlsanity.com/paper/2501.12948/) (2025) - [Measuring Mathematical Problem Solving With the MATH Dataset](https://mlsanity.com/paper/2103.03874/) (2021) - [DAPO: An Open-Source LLM Reinforcement Learning System at Scale](https://mlsanity.com/paper/2503.14476/) (2025) - [HybridFlow: A Flexible and Efficient RLHF Framework](https://mlsanity.com/paper/2409.19256/) (2024) - [Solving Quantitative Reasoning Problems with Language Models](https://mlsanity.com/paper/2206.14858/) (2022) - [OlympiadBench: A Challenging Benchmark for Promoting AGI with Olympiad-Level Bilingual Multimodal Scientific Problems](https://mlsanity.com/paper/2402.14008/) (2024) - [Qwen2.5-Math Technical Report: Toward Mathematical Expert Model via Self-Improvement](https://mlsanity.com/paper/2409.12122/) (2024) - [SimpleRL-Zoo: Investigating and Taming Zero Reinforcement Learning for Open Base Models in the Wild](https://mlsanity.com/paper/2503.18892/) (2025) - [Humanity's Last Exam](https://mlsanity.com/paper/2501.14249/) (2025) - [The Lessons of Developing Process Reward Models in Mathematical Reasoning](https://mlsanity.com/paper/2501.07301/) (2025) - [Towards Reasoning Era: A Survey of Long Chain-of-Thought for Reasoning Large Language Models](https://mlsanity.com/paper/2503.09567/) (2025) - [Skywork-Reward-V2: Scaling Preference Data Curation via Human-AI Synergy](https://mlsanity.com/paper/2507.01352/) (2025) - [A Survey of Reinforcement Learning for Large Reasoning Models](https://mlsanity.com/paper/2509.08827/) (2025) - [AIMO-2 Winning Solution: Building State-of-the-Art Mathematical Reasoning Models with OpenMathReasoning dataset](https://mlsanity.com/paper/2504.16891/) (2025) - [Crossing the Reward Bridge: Expanding RL with Verifiable Rewards Across Diverse Domains](https://mlsanity.com/paper/2503.23829/) (2025) - [A Minimalist Approach to LLM Reasoning: from Rejection Sampling to Reinforce](https://mlsanity.com/paper/2504.11343/) (2025) - [General-Reasoner: Advancing LLM Reasoning Across All Domains](https://mlsanity.com/paper/2505.14652/) (2025) - [OctoThinker: Mid-training Incentivizes Reinforcement Learning Scaling](https://mlsanity.com/paper/2506.20512/) (2025) - [Reinforcement Learning with Rubric Anchors](https://mlsanity.com/paper/2508.12790/) (2025) - [Big-Math: A Large-Scale, High-Quality Math Dataset for Reinforcement Learning in Language Models](https://mlsanity.com/paper/2502.17387/) (2025) - [Leveraging Grammar and Reinforcement Learning for Neural Program Synthesis](https://mlsanity.com/paper/1805.04276/) (2018) - [AceMath: Advancing Frontier Math Reasoning with Post-Training and Reward Modeling](https://mlsanity.com/paper/2412.15084/) (2024) - [Reinforcing General Reasoning without Verifiers](https://mlsanity.com/paper/2505.21493/) (2025) - [MegaScience: Pushing the Frontiers of Post-Training Datasets for Science Reasoning](https://mlsanity.com/paper/2507.16812/) (2025) - [xVerify: Efficient Answer Verifier for Reasoning Model Evaluations](https://mlsanity.com/paper/2504.10481/) (2025) - [Exploring the Limit of Outcome Reward for Learning Mathematical Reasoning](https://mlsanity.com/paper/2502.06781/) (2025) ## Built on by (most cited first, 20 of 20) - [CodeV: Code with Images for Faithful Visual Reasoning via Tool-Aware Policy Optimization](https://mlsanity.com/paper/2511.19661/) (2025) - [RAGEN-2: Reasoning Collapse in Agentic RL](https://mlsanity.com/paper/2604.06268/) (2026) - [PROGRESSLM: Towards Progress Reasoning in Vision-Language Models](https://mlsanity.com/paper/2601.15224/) (2026) - [Certified Self-Consistency: Statistical Guarantees and Test-Time Training for Reliable Reasoning in LLMs](https://mlsanity.com/paper/2510.17472/) (2025) - [Silence the Judge: Reinforcement Learning with Self-Verifier via Latent Geometric Clustering](https://mlsanity.com/paper/2601.08427/) (2026) - [AVSD: Adaptive-View Self-Distillation by Balancing Consensus and Teacher-Specific Privileged Signals](https://mlsanity.com/paper/2605.20643/) (2026) - [SA-VLA: Spatially-Aware Flow-Matching for Vision-Language-Action Reinforcement Learning](https://mlsanity.com/paper/2602.00743/) (2026) - [Reasoning over mathematical objects: on-policy reward modeling and test time aggregation](https://mlsanity.com/paper/2603.18886/) (2026) - [ExecCritic: Learn to Test, Test to Improve for Coding Agents](https://mlsanity.com/paper/2609.09133/) (2026) - [RLVP: Penalize the Path, Reward the Outcome](https://mlsanity.com/paper/2607.07435/) (2026) - [When Errors Can Be Beneficial: A Categorization of Imperfect Rewards for Policy Gradient](https://mlsanity.com/paper/2604.25872/) (2026) - [Coverage, Not Targeting: A Structural Regime in Multi-Turn Agent Credit Assignment](https://mlsanity.com/paper/2609.02417/) (2026) - [Miner:Mining Intrinsic Mastery for Data-Efficient RL in Large Reasoning Models](https://mlsanity.com/paper/2601.04731/) (2026) - [Configurable Reward Model for Balanced Safety Alignment](https://mlsanity.com/paper/2605.30487/) (2026) - [Internalizing Geometric Law: Learning from Solver Residuals for Precision-Critical Generation](https://mlsanity.com/paper/2606.09278/) (2026) - [ProcessThinker: Enhancing Multi-modal Large Language Models Reasoning via Rollout-based Process Reward](https://mlsanity.com/paper/2606.11209/) (2026) - [Enhancing VLM Reward Models Through Structure-Aware Fine-Tuning](https://mlsanity.com/paper/2608.03875/) (2026) - [ARC: Fair Relative Advantage Comparison in Open-Ended Real-World Interaction](https://mlsanity.com/paper/2608.13622/) (2026) - [RLVR landscapes for iterated multiplications can be benign: Insights from spin-glass theory](https://mlsanity.com/paper/2609.28625/) (2026) - [Beyond Binary Verifiable Rewards: Behavioral Self-Rewarding for SWE Agents](https://mlsanity.com/paper/2610.12598/) (2026) ## Abstract (arXiv) Post-training for reasoning of large language models (LLMs) increasingly relies on verifiable rewards: deterministic checkers that provide 0-1 correctness signals. While reliable, such binary feedback is brittle--many tasks admit partially correct or alternative answers that verifiers under-credit, and the resulting all-or-nothing supervision limits learning. Reward models offer richer, continuous feedback, which can serve as a complementary supervisory signal to verifiers. We introduce HERO (Hybrid Ensemble Reward Optimization), a reinforcement learning framework that integrates verifier signals with reward-model scores in a structured way. HERO employs stratified normalization to bound reward-model scores within verifier-defined groups, preserving correctness while refining quality distinctions, and variance-aware weighting to emphasize challenging prompts where dense signals matter most. Across diverse mathematical reasoning benchmarks, HERO consistently outperforms RM-only and verifier-only baselines, with strong gains on both verifiable and hard-to-verify tasks. Our results show that hybrid reward design retains the stability of verifiers while leveraging the nuance of reward models to advance reasoning.