mlsanityData to 14 Oct 2026

Grad2Reward: From Sparse Judgment to Dense Rewards for Improving Open-Ended LLM Reasoning

arXiv 2602.01791 ↗First posted 2 Feb 2026cs.LGcited by 6

On the map: LLM reasoning → Reinforcement learning for reasoning

also in 1 more
Abstract (arXiv)

Reinforcement Learning with Verifiable Rewards (RLVR) has catalyzed significant breakthroughs in complex LLM reasoning within verifiable domains, such as mathematics and programming. Recent efforts have sought to extend this paradigm to open-ended tasks by employing LLMs-as-a-Judge to provide sequence-level rewards for policy optimization. However, these rewards are inherently sparse, failing to provide the fine-grained supervision necessary for generating complex, long-form trajectories. Furthermore, current work treats the Judge as a black-box oracle, discarding the rich intermediate feedback signals encoded in it. To address these limitations, we introduce Grad2Reward, a novel framework that extracts dense process rewards directly from the Judge's model inference process via a single backward pass. By leveraging gradient-based attribution, Grad2Reward enables precise token-level credit assignment, substantially enhancing training efficiency and reasoning quality. Additionally, Grad2Reward introduces a self-judging mechanism, allowing the policy to improve through its own evaluative signals without training specialized reward models or reliance on superior external Judges. The experiments demonstrate that policies optimized with Grad2Reward achieve outstanding performance across diverse open-ended tasks, affirming its effectiveness and broad generalizability.

Key facts. Cited by 6 papers in the map (data through 14 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 18 of its 43 references that are in the map. First posted on arXiv 2 Feb 2026.

Builds on

18 of 43 references.

Closest ancestors · 7

Same direction, newest first

  1. 2025Reward and Guidance through Rubrics: Promoting Exploration to Improve Multi-Domain Reasoning15 Nov
  2. 2025Linking Process to Outcome: Conditional Reward Modeling for LLM Reasoning30 Sep
  3. 2025Self-Rewarding Rubric-Based Reinforcement Learning for Open-Ended Reasoning19 Sep
  4. 2025Breaking the Exploration Bottleneck: Rubric-Scaffolded Reinforcement Learning for General LLM Reasoning23 Aug
  5. 2025Direct Reasoning Optimization: Token-Level Reasoning Reflectivity Meets Rubric Gates for Unverifiable Tasks16 Jun
  6. 2025Process Reinforcement through Implicit Rewards3 Feb
  7. 2025The Lessons of Developing Process Reward Models in Mathematical Reasoning13 Jan

Other references · 11

  1. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  2. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  3. 2024HybridFlow: A Flexible and Efficient RLHF Frameworkcited by 2,057
  4. 2024OpenAI o1 System Cardcited by 1,244
  5. 2025HealthBench: Evaluating Large Language Models Towards Improved Human Healthcited by 429
  6. 2025Rubrics as Rewards: Reinforcement Learning Beyond Verifiable Domainscited by 366
  7. 2025DR Tulu: Reinforcement Learning with Evolving Rubrics for Deep Researchcited by 114
  8. 2025Reinforcement Learning with Rubric Anchorscited by 113
  9. 2025JudgeLRM: Large Reasoning Models as a Judgecited by 78
  10. 2025ResearchQA: Evaluating Scholarly Question Answering at Scale Across 75 Fields with Survey-Mined Questions and Rubricscited by 49
  11. 2025InfiMed-ORBIT: Aligning LLMs on Open-Ended Complex Tasks via Rubric-Based Incremental Trainingcited by 12

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 6.

6 papers.

  1. 2026RecurSE: Bounded Recursive Self-Evaluation for LLM Rubric Judgescited by 1
  2. 2026Gradients Know What Outcomes Don't: Unlocking Reinforcement Learning for LLM Reasoning with Gradient-Aligned Rewardscited by 1
  3. 2026OPERA: Aligning Open-Ended Reasoning via Objective Perplexity-based Reinforcement Learningcited by 0
  4. 2026From Judgment Quality to Downstream Utility: Rethinking LLM-as-a-Judge for Open-Ended Taskscited by 0
  5. 2026Advancing Entropy-Level Credit Assignment in RLVR via Proximal Entropy Policy Optimizationcited by 0
  6. 2026SOAR: Source-Aware Reinforcement Learning with Precise Credit Assignment for Search Agentscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025RLPR: Extrapolating RLVR to General Domains without Verifiers
  2. 2025ConfClip: Confidence-Weighted and Clipped Reward for Reinforcement Learning in LLMs
  3. 2024Rewarding Progress: Scaling Automated Process Verifiers for LLM Reasoning
  4. 2025Reinforcement Learning with Verifiable Rewards Implicitly Incentivizes Correct Reasoning in Base LLMs
  5. 2024On Designing Effective RL Reward at Training Time for LLM Reasoning
  6. 2025No Free Lunch: Rethinking Internal Feedback for LLM Reasoning
  7. 2024Teaching Large Language Models to Reason with Reinforcement Learning
  8. 2026Re2: Unlocking LLM Reasoning via Reinforcement Learning with Re-solving