Exploiting Verification-Generation Gap: Test-Time Reinforcement Learning with Confidence-Conditioned Verification
On the map: LLM reasoning → Reinforcement learning for reasoning
Abstract (arXiv)
Test-time reinforcement learning has emerged as a promising paradigm for enhancing the complex reasoning abilities of large language models in a completely label-free manner. Despite existing studies focusing on Pass@1 performance, optimizing Pass@k remains under-explored yet critical in label-free settings, which measures generation coverage for sustained exploration. Optimizing Pass@k in label-free setting is highly non-trivial, as directly applying the Pass@k advantage designs effective for RLVR yields unsatisfactory performance. Through in-depth empirical analysis, we discover the root causes hindering performance: pseudo-label estimations for low-confidence samples have a high probability of being incorrect, while candidate answers for high-confidence samples suffer from severe diversity collapse. To overcome these hurdles, we propose TTRL-CoCoV (Test-Time Reinforcement Learning with Confidence-Conditioned Verification), a novel confidence-adaptive framework that expands Pass@k coverage and improves Pass@1 performance. Based on our key insight that verification capability generally leads generation capability, TTRL-CoCoV employs a confidence-conditioned mechanism: for high-confidence samples, it bootstraps verifier and applies an exploration-enhancing reward to prevent diversity collapse; for low-confidence samples, it delegates pseudo-label selection to the verifier to filter incorrect pseudo-labels; and for medium-confidence samples, it bypasses verification entirely. Extensive experiments demonstrate that TTRL-CoCoV outperforms the best competing methods across 6 widely-recognized benchmarks, achieves average absolute gains of +9.8% in Pass@1 and +18.7% in Pass@16 over TTRL, and even achieves absolute Pass@1 improvements of up to +5.0% across multiple reasoning benchmarks when compared against fully supervised RL methods. Our code repository: https://github.com/shanjf666/CoCoV.
Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 36 of its 39 references that are in the map. First posted on arXiv 2 Jun 2026.
Builds on
36 of 39 references.
Closest ancestors · 16
Same direction, newest first
- 2026What If Consensus Lies? Selective-Complementary Reinforcement Learning at Test Time20 Mar
- 2026CoVerRL: Breaking the Consensus Trap in Label-Free Reasoning via Generator-Verifier Co-Evolution18 Mar
- 2026Tool Verification for Test-Time Reinforcement Learning2 Mar
- 2026DSDR: Dual-Scale Diversity Regularization for Exploration in LLM Reasoning23 Feb
- 2025SPINE: Token-Selective Test-Time Reinforcement Learning with Entropy-Band Regularization22 Nov
- 2025Better LLM Reasoning via Dual-Play14 Nov
- 2025RESTRAIN: From Spurious Votes to Signals -- Self-Driven RL with Self-Penalization2 Oct
- 2025Outcome-based Exploration for LLM Reasoning8 Sep
- 2025Pass@k Training for Adaptively Balancing Exploration and Exploitation of Large Reasoning Models14 Aug
- 2025Co-rewarding: Stable Self-supervised RL for Eliciting Reasoning in Large Language Models1 Aug
- 2025OctoThinker: Mid-training Incentivizes Reinforcement Learning Scaling25 Jun
- 2025Can Large Reasoning Models Self-Train?27 May
4 more
Other references · 20
- 2024The Llama 3 Herd of Modelscited by 14,192
- 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
- 2025Qwen3 Technical Reportcited by 8,195
- 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
- 2023GPQA: A Graduate-Level Google-Proof Q&A Benchmarkcited by 2,748
- 2024Qwen2.5-Math Technical Report: Toward Mathematical Expert Model via Self-Improvementcited by 1,057
- 2022STaR: Bootstrapping Reasoning With Reasoningcited by 895
- 2024Self-Rewarding Language Modelscited by 654
- 2022Large Language Models are Better Reasoners with Self-Verificationcited by 304
- 2025R-Zero: Self-Evolving Reasoning LLM from Zero Datacited by 210
- 2024Forest-of-Thought: Scaling Test-Time Compute for Enhancing LLM Reasoningcited by 110
- 2025Pass@K Policy Optimization: Solving Harder Reinforcement Learning Problemscited by 83
8 more
- 2025Evolving Language Models without Labels: Majority Drives Selection, Novelty Promotes Variationcited by 44
- 2025SPC: Evolving Self-Play Critic via Adversarial Games for LLM Reasoningcited by 40
- 2025ETTRL: Balancing Exploration and Exploitation in LLM Test-Time Reinforcement Learning Via Entropy Mechanismcited by 35
- 2026Learning to Self-Verify Makes Language Models Better Reasonerscited by 18
- 2025Incentivizing LLMs to Self-Verify Their Answerscited by 16
- 2025Self-Harmony: Learning to Harmonize Self-Supervision and Self-Play in Test-Time Reinforcement Learningcited by 9
- 2025Test-time Diverse Reasoning by Riemannian Activation Steeringcited by 3
- 2026Distribution-Aware Reward Estimation for Test-Time Reinforcement Learningcited by 2
Built on by
Citing papers per year: 2026: 1.
1 paper.
Similar, not cited
Close in wording, no citation either way
- 2026Scaling with Confidence: Calibrating Confidence of LLMs for Adaptive Test Time Scaling
- 2026Hi-TTRL: Regulating Consensus with Hints for Test-Time Reinforcement Learning
- 2026TTVS: Boosting Self-Exploring Reinforcement Learning via Test-time Variational Synthesis
- 2026Process Supervision of Confidence Margin for Calibrated LLM Reasoning
- 2026TTPO: Test-Time Policy Optimization
- 2026SVR: Self-Verifying Refinement via Joint Verdict-Confidence Reinforcement Learning for Adaptive Test-Time Compute
- 2025Beyond Majority Voting: Towards Fine-grained and More Reliable Reward Signal for Test-Time Reinforcement Learning
- 2026ECHO: Entropy-Confidence Hybrid Optimization for Test-Time Reinforcement Learning