mlsanityData to 14 Oct 2026

ETTRL: Balancing Exploration and Exploitation in LLM Test-Time Reinforcement Learning Via Entropy Mechanism

arXiv 2508.11356 ↗First posted 15 Aug 2025cs.LG cs.AIcited by 36

Not on the map. Why?

Cited by 22 in LLM reasoning, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Recent advancements in Large Language Models have yielded significant improvements in complex reasoning tasks such as mathematics and programming. However, these models remain heavily dependent on annotated data and exhibit limited adaptability in unsupervised scenarios. To address these limitations, test-time reinforcement learning (TTRL) has been proposed, which enables self-optimization by leveraging model-generated pseudo-labels. Despite its promise, TTRL faces several key challenges, including high inference costs due to parallel rollouts and early-stage estimation bias that fosters overconfidence, reducing output diversity and causing performance plateaus. To address these challenges, we introduce an entropy-based mechanism to enhance the exploration-exploitation balance in test-time reinforcement learning through two strategies: Entropy-fork Tree Majority Rollout (ETMR) and Entropy-based Advantage Reshaping (EAR). Compared with the baseline, our approach enables Llama3.1-8B to achieve a 68 percent relative improvement in Pass at 1 metric on the AIME 2024 benchmark, while consuming only 60 percent of the rollout tokens budget. This highlights our method's ability to effectively optimize the trade-off between inference efficiency, diversity, and estimation robustness, thereby advancing unsupervised reinforcement learning for open-domain reasoning tasks.

Key facts. Cited by 36 papers in the map (data through 14 Oct 2026). Builds on 20 of its 27 references that are in the map. First posted on arXiv 15 Aug 2025.

Builds on

20 of 27 references.

References · 20

  1. 2017Proximal Policy Optimization Algorithmscited by 15,215
  2. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  3. 2025Qwen3 Technical Reportcited by 8,820
  4. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 6,330
  5. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,708
  6. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,371
  7. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
  8. 2023Let's Verify Step by Stepcited by 3,789
  9. 2023Math-Shepherd: Verify and Reinforce LLMs Step-by-step without Human Annotationscited by 928
  10. 2025Beyond the 80/20 Rule: High-Entropy Minority Tokens Drive Effective Reinforcement Learning for LLM Reasoningcited by 559
  11. 2025The Entropy Mechanism of Reinforcement Learning for Reasoning Language Modelscited by 485
  12. 2025Reasoning with Exploration: An Entropy Perspectivecited by 281
8 more
  1. 2025TTRL: Test-Time Reinforcement Learningcited by 275
  2. 2025Spurious Rewards: Rethinking Training Signals in RLVRcited by 209
  3. 2024SciCode: A Research Coding Benchmark Curated by Scientistscited by 178
  4. 2025The Unreasonable Effectiveness of Entropy Minimization in LLM Reasoningcited by 166
  5. 2025Can 1B LLM Surpass 405B LLM? Rethinking Compute-Optimal Test-Time Scalingcited by 136
  6. 2025Maximizing Confidence Alone Improves Reasoningcited by 100
  7. 2025TreeRL: LLM Reinforcement Learning with On-Policy Tree Searchcited by 80
  8. 2025What Makes a Reward Model a Good Teacher? An Optimization Perspectivecited by 73

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 8; 2026: 28.

36 papers.

Sort:

By area:

  1. 2025A Survey of Reinforcement Learning for Large Reasoning Modelscited by 180
  2. 2025Evolving Language Models without Labels: Majority Drives Selection, Novelty Promotes Variationcited by 45
  3. 2025Agentic Entropy-Balanced Policy Optimizationcited by 40
  4. 2025Diversity-Incentivized Exploration for Versatile Reasoningcited by 28
  5. 2025RESTRAIN: From Spurious Votes to Signals -- Self-Driven RL with Self-Penalizationcited by 15
  6. 2025SPINE: Token-Selective Test-Time Reinforcement Learning with Entropy-Band Regularizationcited by 12
  7. 2026CoVerRL: Breaking the Consensus Trap in Label-Free Reasoning via Generator-Verifier Co-Evolutioncited by 12
  8. 2026What If Consensus Lies? Selective-Complementary Reinforcement Learning at Test Timecited by 12
  9. 2025Self-Harmony: Learning to Harmonize Self-Supervision and Self-Play in Test-Time Reinforcement Learningcited by 10
  10. 2026DSDR: Dual-Scale Diversity Regularization for Exploration in LLM Reasoningcited by 9
  11. 2026PRPO: Aligning Process Reward with Outcome Reward in Policy Optimizationcited by 7
  12. 2026DPWriter: Reinforcement Learning with Diverse Planning Branching for Creative Writingcited by 6
24 more
  1. 2026When Models Judge Themselves: Unsupervised Self-Evolution for Multimodal Reasoningcited by 5
  2. 2026Self-Improvement of Large Language Models: A Technical Overview and Future Outlookcited by 5
  3. 2026Scaling Large Reasoning Models beyond Human Supervision: A Path toward Superintelligencecited by 5
  4. 2026ECHO: Entropy-Confidence Hybrid Optimization for Test-Time Reinforcement Learningcited by 4
  5. 2026Understanding and Mitigating Spurious Signal Amplification in Test-Time Reinforcement Learning for Math Reasoningcited by 3
  6. 2025Entropy-Guided Loop: Achieving Reasoning through Uncertainty-Aware Generationcited by 2
  7. 2026Dual Consensus: Escaping from Spurious Majority in Unsupervised RLVR via Two-Stage Vote Mechanismcited by 2
  8. 2026Controllable Exploration in Hybrid-Policy RLVR for Multi-Modal Reasoningcited by 1
  9. 2026ConsistRM: Improving Generative Reward Models via Consistency-Aware Self-Trainingcited by 1
  10. 2026Breaking textitWinner-Takes-All: Cooperative Policy Optimization Improves Diverse LLM Reasoningcited by 1
  11. 2026Exploiting Verification-Generation Gap: Test-Time Reinforcement Learning with Confidence-Conditioned Verificationcited by 1
  12. 2026GeoMin: Data-Efficient Semi-Supervised RLVR via Geometric Distribution Modelingcited by 1
  13. 2026Continual Test-Time Adaptation in Computer Vision: Methods, Benchmarks, and Future Directionscited by 1
  14. 2026Hi-TTRL: Regulating Consensus with Hints for Test-Time Reinforcement Learningcited by 1
  15. 2026Agentic-TTT: Training test-time policy for test-time trainingcited by 1
  16. 2026MAPLE: Elevating Medical Reasoning from Statistical Consensus to Process-Led Alignmentcited by 0
  17. 2026DiPO: Disentangled Perplexity Policy Optimization for Fine-grained Exploration-Exploitation Trade-Offcited by 0
  18. 2026Free Energy-Driven Reinforcement Learning with Adaptive Advantage Shaping for Unsupervised Reasoning in LLMscited by 0
  19. 2026When Do Intrinsic Rewards Work for Code Reasoning? A Comprehensive Studycited by 0
  20. 2026Unsupervised Post-Training of Foundation Models: A Surveycited by 0
  21. 2026Rewarding Reasoning, Not Answers: Fixing and Bounding Test-Time Reinforcement Learning on Medical QAcited by 0
  22. 2026Listen Then Reason: Perception-Grounded Test-Time Reinforcement Learning for Large Audio-Language Modelscited by 0
  23. 2026Sharpen Before You Adapt: Data-Free Entry-State Sharpening for Test-Time Reinforcement Learningcited by 0
  24. 2026SelectTTRL: Representation-Guided Data Selection for Efficient Test-Time Reinforcement Learningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Addressing Performance Saturation for LLM RL via Precise Entropy Curve Control
  2. 2026Targeted Exploration via Unified Entropy Control for Reinforcement Learning
  3. 2025Revisiting Entropy Regularization: Adaptive Coefficient Unlocks Its Potential for LLM Reinforcement Learning
  4. 2025Beyond High-Entropy Exploration: Correctness-Aware Low-Entropy Segment-Based Advantage Shaping for Reasoning LLMs
  5. 2026Entropy-Preserving Reinforcement Learning
  6. 2026Surprising Success, Repeated Failure: Entropy-Guided Credit Assignment for Exploration in LLM Reasoning
  7. 2026Tool-Aware Optimization with Entropy Guidance for Efficient Agentic Reinforcement Learning
  8. 2026Back to Basics: Revisiting Exploration in Reinforcement Learning for LLM Reasoning via Generative Probabilities