# Detecting and Mitigating the Correct-Answer Extinction Window in Test-Time Reinforcement Learning with Majority Voting

arXiv: https://arxiv.org/abs/2605.19444 · HTML: https://mlsanity.com/paper/2605.19444/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning.
- Builds on 14 of its 20 references that are in the map.
- First posted on arXiv 19 May 2026.

## Placed in

- LLM reasoning / [Reinforcement learning for reasoning](https://mlsanity.com/lr/reinforcement-learning-for-reasoning/)

## Builds on (most cited first, 14 of 14)

- [Measuring Mathematical Problem Solving With the MATH Dataset](https://mlsanity.com/paper/2103.03874/) (2021)
- [Learning to Reason without External Rewards](https://mlsanity.com/paper/2505.19590/) (2025)
- [The Unreasonable Effectiveness of Entropy Minimization in LLM Reasoning](https://mlsanity.com/paper/2505.15134/) (2025)
- [Right Question is Already Half the Answer: Fully Unsupervised LLM Reasoning Incentivization](https://mlsanity.com/paper/2504.05812/) (2025)
- [Evolving Language Models without Labels: Majority Drives Selection, Novelty Promotes Variation](https://mlsanity.com/paper/2509.15194/) (2025)
- [Reinforcement Pre-Training](https://mlsanity.com/paper/2506.08007/) (2025)
- [Scalable Power Sampling: Unlocking Efficient, Training-Free Reasoning for LLMs via Distribution Sharpening](https://mlsanity.com/paper/2601.21590/) (2026)
- [LADDER: Self-Improving LLMs Through Recursive Problem Decomposition](https://mlsanity.com/paper/2503.00735/) (2025)
- [TTCS: Test-Time Curriculum Synthesis for Self-Evolving](https://mlsanity.com/paper/2601.22628/) (2026)
- [Beyond Majority Voting: Towards Fine-grained and More Reliable Reward Signal for Test-Time Reinforcement Learning](https://mlsanity.com/paper/2512.15146/) (2025)
- [CoVerRL: Breaking the Consensus Trap in Label-Free Reasoning via Generator-Verifier Co-Evolution](https://mlsanity.com/paper/2603.17775/) (2026)
- [Tool Verification for Test-Time Reinforcement Learning](https://mlsanity.com/paper/2603.02203/) (2026)
- [Meta-TTRL: A Metacognitive Framework for Self-Improving Test-Time Reinforcement Learning in Unified Multimodal Models](https://mlsanity.com/paper/2603.15724/) (2026)
- [DiSCTT: Consensus-Guided Self-Curriculum for Efficient Test-Time Adaptation in Reasoning](https://mlsanity.com/paper/2603.05357/) (2026)

## Built on by (most cited first, 7 of 7)

- [Scaling Large Reasoning Models beyond Human Supervision: A Path toward Superintelligence](https://mlsanity.com/paper/2608.31075/) (2026)
- [SERPO: Self-Evolving Rubric Policy Optimization for Open-Ended Test-Time Reinforcement Learning](https://mlsanity.com/paper/2607.26873/) (2026)
- [Phantom Gains: Auditing Self-Improvement Against a Measured Null](https://mlsanity.com/paper/2608.20290/) (2026)
- [TTPO: Test-Time Policy Optimization](https://mlsanity.com/paper/2608.27448/) (2026)
- [Harnessing Image Question Dependence for Better VLM Test-time Reinforcement Learning](https://mlsanity.com/paper/2609.13296/) (2026)
- [SelectTTRL: Representation-Guided Data Selection for Efficient Test-Time Reinforcement Learning](https://mlsanity.com/paper/2610.16104/) (2026)
- [Checking the Checkers: A Synthesis of Verifier Noise and Robustness in RLVR](https://mlsanity.com/paper/2610.17824/) (2026)

## Abstract (arXiv)

Test-time reinforcement learning (TTRL) reports substantial accuracy gains on mathematical reasoning benchmarks using majority vote as a pseudo-label signal. We argue these gains are systematically misinterpreted: most reflect sharpening of already-solvable problems rather than genuine learning, while problems corrupted from correct to incorrect outnumber truly learned ones, and this damage is irreversible once majority vote locks onto a wrong answer. Per-problem tracking reveals that correct-answer signals in low-ability problems are briefly active before being permanently suppressed, a phenomenon we term the textitCorrect-Answer Extinction Window, with Flip Rate (FR) as its leading indicator. We thus propose TTRL-Guard, a lightweight framework with three mechanisms targeting the extinction window: Flip-Rate-Aware Reward Scaling (FRS) down-weights at-risk updates as FR declines, Minority-Preserving Sampling (MPS) retains gradient signal from minority correct answers, and Risk-Conditioned Sparse Updatings (RCSU) suspends updates on polarized problems. Experiments across three models and four benchmarks show that TTRL-Guard achieves the best average pass@1 on Qwen2.5-7B-Instruct and Qwen3-4B, improves relatively over TTRL by +54% on AIME 2025.