mlsanityData to 14 Oct 2026

Tool Verification for Test-Time Reinforcement Learning

arXiv 2603.02203 ↗First posted 2 Mar 2026cs.AI cs.CLcited by 9

On the map: LLM reasoning → Reinforcement learning for reasoning

also in 1 more
Abstract (arXiv)

Test-time reinforcement learning (TTRL) has emerged as a promising paradigm for Recursive Self-Improving AI (RSI) by adapting Large Reasoning Models (LRMs) on unlabeled test inputs, using self-consensus rewards derived from majority voting over sampled rollouts. However, majority voting can mistake popularity for correctness: a spurious yet high-frequency unverified consensus may become a biased reward signal, causing test-time RL to reinforce frequent but wrong answers and collapse into an incorrect mode. We address this false-popular failure mode with T3RL (Tool-Verification for Test-Time Reinforcement Learning), a verification-aware test-time RL framework. T3RL grounds pseudo-label construction in external tool evidence. Concretely, a verifier utilizes external tool evidence (e.g., from code execution) to upweight verified rollouts during a verification-aware voting, producing more reliable pseudo-labels for training. Across various math difficulties (MATH-500, AMC, and AIME 2024) and diverse backbone families, T3RL significantly improves over TTRL, with better performance on harder problems. T3RL is positioned as a verified online data synthesizer, highlighting the role of tool verification in reliable online adaptation. T3RL is extensible to verifiable domains and provides training and inference efficiency.

Key facts. Cited by 9 papers in the map (data through 14 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 39 of its 47 references that are in the map. First posted on arXiv 2 Mar 2026.

Builds on

39 of 47 references.

Closest ancestors · 9

Same direction, newest first

  1. 2025Spurious Rewards: Rethinking Training Signals in RLVR12 Jun
  2. 2025Consistent Paths Lead to Truth: Self-Rewarding Reinforcement Learning for LLM Reasoning10 Jun
  3. 2025Reinforcement Learning for Reasoning in Large Language Models with One Training Example29 Apr
  4. 2025TTRL: Test-Time Reinforcement Learning22 Apr
  5. 2025SimpleRL-Zoo: Investigating and Taming Zero Reinforcement Learning for Open Base Models in the Wild24 Mar
  6. 2024Generative Verifiers: Reward Modeling as Next-Token Prediction27 Aug
  7. 2023Math-Shepherd: Verify and Reinforce LLMs Step-by-step without Human Annotations14 Dec
  8. 2023Let's Verify Step by Step31 May
  9. 2022Solving math word problems with process- and outcome-based feedback25 Nov

Other references · 30

  1. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  2. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  3. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,708
  4. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,371
  5. 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,182
  6. 2024Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameterscited by 1,986
  7. 2025s1: Simple test-time scalingcited by 1,341
  8. 2025ReTool: Reinforcement Learning for Strategic Tool Use in LLMscited by 392
  9. 2024Learning to (Learn at Test Time): RNNs with Expressive Hidden Statescited by 355
  10. 2024Titans: Learning to Memorize at Test Timecited by 328
  11. 2024Inference Scaling Laws: An Empirical Analysis of Compute-Optimal Inference for Problem-Solving with Language Modelscited by 290
  12. 2023ToRA: A Tool-Integrated Reasoning Agent for Mathematical Problem Solvingcited by 282
18 more
  1. 2024LLMs Know More Than They Show: On the Intrinsic Representation of LLM Hallucinationscited by 244
  2. 2024The pitfalls of next-token predictioncited by 160
  3. 2023Deductive Verification of Chain-of-Thought Reasoningcited by 124
  4. 2024On Memorization of Large Language Models in Logical Reasoningcited by 117
  5. 2026Learning to Discover at Test Timecited by 115
  6. 2025One-Minute Video Generation with Test-Time Trainingcited by 101
  7. 2024The Surprising Effectiveness of Test-Time Training for Few-Shot Learningcited by 96
  8. 2024Buffer of Thoughts: Thought-Augmented Reasoning with Large Language Modelscited by 91
  9. 2023Test-Time Training on Nearest Neighbors for Large Language Modelscited by 88
  10. 2025Multi-Agent Verification: Scaling Test-Time Compute with Multiple Verifierscited by 62
  11. 2024Self-Consistency Preference Optimizationcited by 50
  12. 2023Test-Time Training on Video Streamscited by 37
  13. 2025Two Heads are Better Than One: Test-time Scaling of Multi-agent Collaborative Reasoningcited by 29
  14. 2024TOOLVERIFIER: Generalization to New Tools via Self-Verificationcited by 24
  15. 2024Montessori-Instruct: Generate Influential Training Data Tailored for Student Learningcited by 13
  16. 2025T1: Tool-integrated Verification for Test-time Compute Scaling in Small Language Modelscited by 13
  17. 2025Step-level Verifier-guided Hybrid Test-Time Scaling for Large Language Modelscited by 10
  18. 2025Trust but Verify! A Survey on Verification Design for Test-time Scalingcited by 8

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 9.

9 papers.

  1. 2026Detecting and Mitigating the Correct-Answer Extinction Window in Test-Time Reinforcement Learning with Majority Votingcited by 7
  2. 2026When Self-Belief Misleads: Active Label Acquisition for Reinforcement Learning with Verifiable Rewardscited by 6
  3. 2026A Survey on Self-Improving Test-Time Intelligence: Feedback-Driven Adapting, Learning, and Scaling at Inferencecited by 3
  4. 2026Exploiting Verification-Generation Gap: Test-Time Reinforcement Learning with Confidence-Conditioned Verificationcited by 1
  5. 2026The Digital Apprentice: A Framework for Human-Directed Agentic AI Developmentcited by 1
  6. 2026StarOR: Synergizing Tree Search and Test-Time Reinforcement Learning for Optimization Modelingcited by 1
  7. 2026Unsupervised Post-Training of Foundation Models: A Surveycited by 0
  8. 2026Entropy-Regularized Rank-Masked Policy Optimization for Test-Time Reinforcement Learning in Code Generationcited by 0
  9. 2026Checking the Checkers: A Synthesis of Verifier Noise and Robustness in RLVRcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Hi-TTRL: Regulating Consensus with Hints for Test-Time Reinforcement Learning
  2. 2025Beyond Majority Voting: Towards Fine-grained and More Reliable Reward Signal for Test-Time Reinforcement Learning
  3. 2026TTPO: Test-Time Policy Optimization
  4. 2026What If Consensus Lies? Selective-Complementary Reinforcement Learning at Test Time
  5. 2026CoRE: Consensus Rewards via Equilibrium for Test-Time Reinforcement Learning
  6. 2026SVR: Self-Verifying Refinement via Joint Verdict-Confidence Reinforcement Learning for Adaptive Test-Time Compute
  7. 2026Controllable and Verifiable Tool-Use Data Synthesis for Agentic Reinforcement Learning
  8. 2025Putting the Value Back in RL: Better Test-Time Scaling by Unifying LLM Reasoners With Verifiers