mlsanityData to 14 Oct 2026

Beyond Majority Voting: Towards Fine-grained and More Reliable Reward Signal for Test-Time Reinforcement Learning

arXiv 2512.15146 ↗First posted 17 Dec 2025cs.CLcited by 13

On the map: LLM reasoning → Reinforcement learning for reasoning

also in 1 more
Abstract (arXiv)

Test-time reinforcement learning mitigates the reliance on annotated data by using majority voting results as pseudo-labels, emerging as a complementary direction to reinforcement learning with verifiable rewards (RLVR) for improving reasoning ability. However, this voting strategy often induces confirmation bias and suffers from sparse rewards, limiting the overall performance. In this work, we propose subgroup-specific step-wise confidence-weighted pseudo-label estimation (SCOPE), a framework integrating model confidence and dynamic subgroup partitioning to address these issues. Specifically, SCOPE integrates the proposed step-wise confidence into pseudo label estimation, prioritizing high-quality reasoning paths over simple frequency count. Furthermore, it dynamically partitions the candidate outputs pool into independent subgroups by balancing reasoning quality against exploration diversity. By deriving local consensus via repeat sampling for each sub group, SCOPE provides diverse supervision targets to encourage broader exploration. We conduct experiments across various models and benchmarks, experimental results show that SCOPE consistently outperforms recent baselines. Notably, SCOPE achieving relative improvements of 13.1% on challenging AIME 2025 and 8.1% on AMC. The code is released at https://github.com/szu-tera/SCOPE.

Key facts. Cited by 13 papers in the map (data through 14 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 26 of its 45 references that are in the map. First posted on arXiv 17 Dec 2025.

Builds on

26 of 45 references.

Closest ancestors · 9

Same direction, newest first

  1. 2025Towards High Data Efficiency in Reinforcement Learning with Verifiable Reward1 Sep
  2. 2025RLPR: Extrapolating RLVR to General Domains without Verifiers23 Jun
  3. 2025Reinforcement Learning with Verifiable Rewards Implicitly Incentivizes Correct Reasoning in Base LLMs17 Jun
  4. 2025Beyond the 80/20 Rule: High-Entropy Minority Tokens Drive Effective Reinforcement Learning for LLM Reasoning2 Jun
  5. 2025Learning to Reason without External Rewards26 May
  6. 2025Reinforcement Learning for Reasoning in Large Language Models with One Training Example29 Apr
  7. 2025Crossing the Reward Bridge: Expanding RL with Verifiable Rewards Across Diverse Domains31 Mar
  8. 2025SimpleRL-Zoo: Investigating and Taming Zero Reinforcement Learning for Open Base Models in the Wild24 Mar
  9. 2023Let's Verify Step by Step31 May

Other references · 17

  1. 2024The Llama 3 Herd of Modelscited by 14,715
  2. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  3. 2025Qwen3 Technical Reportcited by 8,820
  4. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  5. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,371
  6. 2024HybridFlow: A Flexible and Efficient RLHF Frameworkcited by 2,057
  7. 2024Qwen2.5-Math Technical Report: Toward Mathematical Expert Model via Self-Improvementcited by 1,090
  8. 2019Pseudo-Labeling and Confirmation Bias in Deep Semi-Supervised Learningcited by 510
  9. 2025AdaCoT: Pareto-Optimal Adaptive Chain-of-Thought Triggering via Reinforcement Learningcited by 87
  10. 2024Panacea: Pareto Alignment via Preference Adaptation for LLMscited by 80
  11. 2020SENTRY: Selective Entropy Optimization via Committee Consistency for Unsupervised Domain Adaptationcited by 78
  12. 2025Evolving Language Models without Labels: Majority Drives Selection, Novelty Promotes Variationcited by 45
5 more
  1. 2025Beyond Correctness: Harmonizing Process and Outcome Rewards through RL Trainingcited by 29
  2. 2025Post-Training Large Language Models via Reinforcement Learning from Self-Feedbackcited by 15
  3. 2025RLAC: Reinforcement Learning with Adversarial Critic for Free-Form Generation Taskscited by 13
  4. 2025The Self-Improvement Paradox: Can Language Models Bootstrap Reasoning Capabilities without External Scaffolding?cited by 7
  5. 2025Improving Rationality in the Reasoning Process of Language Models through Self-playing Gamecited by 7

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 13.

13 papers.

Sort:

By area:

  1. 2026Dynamics Within Latent Chain-of-Thought: An Empirical Study of Causal Structurecited by 11
  2. 2026Detecting and Mitigating the Correct-Answer Extinction Window in Test-Time Reinforcement Learning with Majority Votingcited by 7
  3. 2026When Self-Belief Misleads: Active Label Acquisition for Reinforcement Learning with Verifiable Rewardscited by 6
  4. 2026Distribution-Aware Reward Estimation for Test-Time Reinforcement Learningcited by 3
  5. 2026SAT: Balancing Reasoning Accuracy and Efficiency with Stepwise Adaptive Thinkingcited by 2
  6. 2026ConsistRM: Improving Generative Reward Models via Consistency-Aware Self-Trainingcited by 1
  7. 2026Hi-TTRL: Regulating Consensus with Hints for Test-Time Reinforcement Learningcited by 1
  8. 2026From the Inside Out: Progressive Distribution Refinement for Confidence Calibrationcited by 0
  9. 2026Unsupervised Post-Training of Foundation Models: A Surveycited by 0
  10. 2026Harnessing Image Question Dependence for Better VLM Test-time Reinforcement Learningcited by 0
  11. 2026Test-time Reinforcement Learning for Anomalous Video Understandingcited by 0
  12. 2026Group-Marginalized Self-Rewarding RL Drives Zero-Label Self-Evolvingcited by 0
1 more
  1. 2026SelectTTRL: Representation-Guided Data Selection for Efficient Test-Time Reinforcement Learningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026CoRE: Consensus Rewards via Equilibrium for Test-Time Reinforcement Learning
  2. 2026Tool Verification for Test-Time Reinforcement Learning
  3. 2026What If Consensus Lies? Selective-Complementary Reinforcement Learning at Test Time
  4. 2026TTPO: Test-Time Policy Optimization
  5. 2026Label-free steering: Compressing test-time reinforcement learning into bias-only subspaces
  6. 2026ECHO: Entropy-Confidence Hybrid Optimization for Test-Time Reinforcement Learning
  7. 2026Exploiting Verification-Generation Gap: Test-Time Reinforcement Learning with Confidence-Conditioned Verification
  8. 2025Rewarding the Journey, Not Just the Destination: A Composite Path and Answer Self-Scoring Reward Mechanism for Test-Time Reinforcement Learning