mlsanityData to 7 Oct 2026

Improving Math Reasoning through Value-guided Informative Search

arXiv 2610.01080 ↗First posted 1 Oct 2026cs.AIcited by 0

On the map: LLM reasoning → Reinforcement learning for reasoning

Abstract (arXiv)

Reinforcement learning with verifiable rewards (RLVR) has substantially improved the mathematical reasoning capabilities of large language models. Recent work introduces search into RLVR rollouts to increase trajectory diversity, but diversity alone does not ensure that the search-induced rollout policy improves upon the current policy. To address this gap, we propose APIVIS, a training-time framework that adapts finite-budget Gumbel search to chunk-level mathematical reasoning. APIVIS combines direct and searched responses within each rollout group, allowing improvements found by search to produce informative relative rewards. It further applies selective supervision to search-improved tokens, preserving a learning signal when uniform group rewards render GRPO ineffective. We show that exact value-guided selection improves the expected verifier reward at each searched state and that this guarantee extends to the complete rollout policy, with a corresponding approximate guarantee under bounded value-estimation error. Experiments on widely recognized mathematical reasoning benchmarks and different model scales demonstrate substantial improvements over competitive search-based methods, validating the effectiveness of APIVIS.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 14 of its 16 references that are in the map. First posted on arXiv 1 Oct 2026.

Builds on

14 of 16 references.

Closest ancestors · 8

Same direction, newest first

  1. 2026Boosting LLM Exploration via Weak-Model Guidance in RLVR27 Aug
  2. 2026Nudging Beyond the Comfort Zone: Efficient Strategy-Guided Exploration for RLVR15 May
  3. 2026Reinforced Efficient Reasoning via Semantically Diverse Exploration8 Jan
  4. 2025Lookahead Tree-Based Rollouts for Enhanced Trajectory-Level Exploration in Reinforcement Learning with Verifiable Rewards28 Oct
  5. 2025Reinforcement Learning for Reasoning in Small LLMs: What Works and What Doesn't20 Mar
  6. 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scale18 Mar
  7. 2024ReST-MCTS*: LLM Self-Training via Process Reward Guided Tree Search6 Jun
  8. 2023Let's Verify Step by Step31 May

Other references · 6

  1. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
  2. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
  3. 2024OlympiadBench: A Challenging Benchmark for Promoting AGI with Olympiad-Level Bilingual Multimodal Scientific Problemscited by 1,099
  4. 2023Alphazero-like Tree-Search can Guide Large Language Model Decoding and Trainingcited by 291
  5. 2025Value-Guided Search for Efficient Chain-of-Thought Reasoningcited by 12
  6. 2026GAUGE: When Not to Trust LLM-as-a-Judge in User-Simulated Evaluation of Task-Oriented Agentscited by 3

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026ReDiPPO: Reference-Guided Value Calibration and Discrepancy-Aware Token Reweighting for Mathematical Reasoning
  2. 2026Offline Exploration-Aware Fine-Tuning for Long-Chain Mathematical Reasoning
  3. 2025Reinforcement Learning for Reasoning in Large Language Models with One Training Example
  4. 2025Limits of Generalization in RLVR: Two Case Studies in Mathematical Reasoning
  5. 2026Learning to Use Tools: Reinforcement Learning for Tool-Integrated Mathematical Reasoning
  6. 2025Online Difficulty Filtering for Reasoning Oriented Reinforcement Learning
  7. 2025Unlocking Reasoning Capabilities in LLMs via Reinforcement Learning Exploration
  8. 2025Act Only When It Pays: Efficient Reinforcement Learning for LLM Reasoning via Selective Rollouts