mlsanityData to 7 Oct 2026

Sharpen Before You Adapt: Data-Free Entry-State Sharpening for Test-Time Reinforcement Learning

arXiv 2610.00903 ↗First posted 1 Oct 2026cs.LGcited by 0

Not on the map. Why?

Abstract (arXiv)

Test-time reinforcement learning (TTRL) adapts language models on unlabeled test problems using supervision derived from their own samples. This makes the checkpoint's entry state consequential: a diffuse policy provides noisier self-supervision and may spend much of a limited adaptation budget merely concentrating probability mass before reliably expressing capability it already possesses. We propose entry-state sharpening: use data-free training before TTRL to prepare a general-purpose checkpoint in a state that subsequent label-free adaptation can exploit more efficiently. The idea is not tied to one training recipe; different data-free objectives can move the same base model to different entry states. Across five data-free checkpoints derived from Qwen3-4B and evaluated under an identical 15-step TTRL protocol, entry policy entropy strongly rank-orders endpoint conversion efficiency, a reliability-to-reachability measure (Spearman ρ=-0.90; ρ=-0.99 after controlling for entry reachability). The contrast across objectives is striking: R-Zero remains diffuse at 3.39 nats and finishes below the untuned base in 6/6 matched comparisons across MATH, GPQA, and AMC, whereas SPIRAL reaches 0.07 nats and achieves the highest post-TTRL accuracy on MATH and GPQA despite its self-play stage using no math training data. An in-domain label-free self-distillation intervention further shows that the entry state can be deliberately sharpened. These results motivate treating checkpoint preparation as a state-control problem: use data-free training to improve TTRL readiness, with entry entropy as a label-free control signal and reachable capability as the constraint.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 33 of its 37 references that are in the map. First posted on arXiv 1 Oct 2026.

Builds on

33 of 37 references.

References · 33

  1. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
  2. 2025Qwen3 Technical Reportcited by 8,195
  3. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,601
  4. 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scalecited by 2,678
  5. 2020Don't Stop Pretraining: Adapt Language Models to Domains and Taskscited by 1,548
  6. 2025Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model?cited by 948
  7. 2022Large Language Models Can Self-Improvecited by 656
  8. 2023Reinforced Self-Training (ReST) for Language Modelingcited by 486
  9. 2025The Entropy Mechanism of Reinforcement Learning for Reasoning Language Modelscited by 463
  10. 2023Scaling Relationship on Learning Mathematical Reasoning with Large Language Modelscited by 417
  11. 2025Cognitive Behaviors that Enable Self-Improving Reasoners, or, Four Habits of Highly Effective STaRscited by 373
  12. 2025Absolute Zero: Reinforced Self-play Reasoning with Zero Datacited by 323
21 more
  1. 2025TTRL: Test-Time Reinforcement Learningcited by 253
  2. 2025R-Zero: Self-Evolving Reasoning LLM from Zero Datacited by 210
  3. 2025Learning to Reason without External Rewardscited by 199
  4. 2025Spurious Rewards: Rethinking Training Signals in RLVRcited by 199
  5. 2025ProRL: Prolonged Reinforcement Learning Expands Reasoning Boundaries in Large Language Modelscited by 187
  6. 2025The Unreasonable Effectiveness of Entropy Minimization in LLM Reasoningcited by 160
  7. 2025Right Question is Already Half the Answer: Fully Unsupervised LLM Reasoning Incentivizationcited by 124
  8. 2025OctoThinker: Mid-training Incentivizes Reinforcement Learning Scalingcited by 110
  9. 2024Self-Improvement in Language Models: The Sharpening Mechanismcited by 102
  10. 2025Maximizing Confidence Alone Improves Reasoningcited by 97
  11. 2024Preserving Diversity in Supervised Fine-Tuning of Large Language Modelscited by 68
  12. 2025SeRL: Self-Play Reinforcement Learning for Large Language Models with Limited Datacited by 50
  13. 2025ETTRL: Balancing Exploration and Exploitation in LLM Test-Time Reinforcement Learning Via Entropy Mechanismcited by 35
  14. 2025One-shot Entropy Minimizationcited by 32
  15. 2025SPINE: Token-Selective Test-Time Reinforcement Learning with Entropy-Band Regularizationcited by 11
  16. 2026What If Consensus Lies? Selective-Complementary Reinforcement Learning at Test Timecited by 10
  17. 2025Reinforcement Mid-Trainingcited by 6
  18. 2026On-Policy Self-Distillation with Sampled Demonstrations Reduces Output Diversitycited by 5
  19. 2026ECHO: Entropy-Confidence Hybrid Optimization for Test-Time Reinforcement Learningcited by 4
  20. 2026Why Do Reasoning Models Lose Coverage? The Role of Data and Forks in the Roadcited by 4
  21. 2026Exploiting Verification-Generation Gap: Test-Time Reinforcement Learning with Confidence-Conditioned Verificationcited by 1

Built on by

Not cited yet.