mlsanityData to 14 Oct 2026

Enhanced LLM Reasoning by Optimizing Reward Functions with Search-Driven Reinforcement Learning

arXiv 2605.02073 ↗First posted 3 May 2026cs.CLcited by 0

On the map: LLM reasoning → Reinforcement learning for reasoning

Abstract (arXiv)

Mathematical reasoning is a key benchmark for large language models. Reinforcement learning is a standard post-training mechanism for improving the reasoning capabilities of large language models, yet performance remains sensitive to the design of the reward function that drives policy optimization. This paper introduces a search-driven framework that treats the reward specification itself as an object of optimization. The setting of interest is one in which the base model is held fixed and the reward specification is the primary remaining design lever. Candidate reward functions are generated by a frontier language model, validated automatically, screened through 500-step Group Relative Policy Optimization (GRPO) training runs on a Llama-3.2-3B-Instruct base model with Low-Rank Adaptation (LoRA), and ranked by F1 on the GSM8K test set. Ranked summaries from prior rounds are then fed back into the next round of generation. Over five rounds, the search produces 50 candidate rewards. The mean F1 rises from 0.596 in Round 1 to 0.632 in Round 5, and the top individual reward reaches F1 = 0.787. Seven ensemble configurations of top-ranked rewards are evaluated. The best ensemble achieves F1 = 0.795 (95% bootstrap CI [0.756, 0.832]) and accuracy 0.660 [0.635, 0.686], a 0.19 absolute F1 gain over a base-rewards-only GRPO baseline (F1 = 0.609). Pairwise McNemar tests with Bonferroni correction show all five-or-more-reward configurations are statistically indistinguishable at α = 0.05/21. A three-seed re-training of the best ensemble yields F1 of 0.785. A randomly drawn 5-reward control collapses to F1 = 0.047, which shows that the ranked-feedback loop, not the additive signal of having more rewards, drives the gain.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 27 of its 33 references that are in the map. First posted on arXiv 3 May 2026.

Builds on

27 of 33 references.

Closest ancestors · 6

Same direction, newest first

  1. 2025Logic-RL: Unleashing LLM Reasoning with Rule-Based Reinforcement Learning20 Feb
  2. 2025The Lessons of Developing Process Reward Models in Mathematical Reasoning13 Jan
  3. 2023Let's Verify Step by Step31 May
  4. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Model29 May
  5. 2022Constitutional AI: Harmlessness from AI Feedback15 Dec
  6. 2017Proximal Policy Optimization Algorithms20 Jul

Other references · 21

  1. 2022Training language models to follow instructions with human feedbackcited by 15,971
  2. 2024The Llama 3 Herd of Modelscited by 14,715
  3. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  4. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  5. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  6. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  7. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,708
  8. 2017Deep reinforcement learning from human preferencescited by 3,912
  9. 2024Self-Rewarding Language Modelscited by 681
  10. 2023Eureka: Human-Level Reward Design via Coding Large Language Modelscited by 571
  11. 2024RewardBench: Evaluating Reward Models for Language Modelingcited by 499
  12. 2023Language to Rewards for Robotic Skill Synthesiscited by 312
9 more
  1. 2023Reward Design with Language Modelscited by 261
  2. 2017Inverse Reward Designcited by 249
  3. 2023Text2Reward: Reward Shaping with Language Models for Reinforcement Learningcited by 160
  4. 2018On Learning Intrinsic Rewards for Policy Gradient Methodscited by 102
  5. 2018Evolved Policy Gradientscited by 88
  6. 2022Automated Reinforcement Learning (AutoRL): A Survey and Open Problemscited by 70
  7. 2024A Large Language Model-Driven Reward Design Framework via Dynamic Feedback for Reinforcement Learningcited by 30
  8. 2022EAGER: Asking and Answering Questions for Automatic Reward Shaping in Language-guided RLcited by 23
  9. 2024REvolve: Reward Evolution with Large Language Models using Human Feedbackcited by 23

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Enhancing Large Language Model Reasoning with Reward Models: An Analytical Survey
  2. 2026Likelihood-Based Reward Designs for General LLM Reasoning
  3. 2025DGRO: Enhancing LLM Reasoning via Exploration-Exploitation Control and Reward Variance Management
  4. 2025Rectifying LLM Thought from Lens of Optimization
  5. 2026GRPO and Reflection Reward for Mathematical Reasoning in Large Language Models
  6. 2026Reward Modeling for Reinforcement Learning-Based LLM Reasoning: Design, Challenges, and Evaluation
  7. 2026Back to Basics: Revisiting Exploration in Reinforcement Learning for LLM Reasoning via Generative Probabilities
  8. 2025Accelerating RL for LLM Reasoning with Optimal Advantage Regression