mlsanityData to 14 Oct 2026

TTRL: Test-Time Reinforcement Learning

arXiv 2504.16084 ↗First posted 22 Apr 2025cs.CL cs.LGcited by 275

On the map: LLM reasoning → Reinforcement learning for reasoning

also in 1 more
Abstract (arXiv)

This paper investigates Reinforcement Learning (RL) on data without explicit labels for reasoning tasks in Large Language Models (LLMs). The core challenge of the problem is reward estimation during inference while not having access to ground-truth information. While this setting appears elusive, we find that common practices in Test-Time Scaling (TTS), such as majority voting, yield surprisingly effective rewards suitable for driving RL training. In this work, we introduce Test-Time Reinforcement Learning (TTRL), a novel method for training LLMs using RL on unlabeled data. TTRL enables self-evolution of LLMs by utilizing the priors in the pre-trained models. Our experiments demonstrate that TTRL consistently improves performance across a variety of tasks and models. Notably, TTRL boosts the pass@1 performance of Qwen-2.5-Math-7B by approximately 211% on the AIME 2024 with only unlabeled test data. Furthermore, although TTRL is only supervised by the maj@n metric, TTRL has demonstrated performance to consistently surpass the upper limit of the initial model maj@n, and approach the performance of models trained directly on test data with ground-truth labels. Our experimental findings validate the general effectiveness of TTRL across various tasks and highlight TTRL's potential for broader tasks and domains. GitHub: https://github.com/PRIME-RL/TTRL

Key facts. Cited by 275 papers in the map (data through 14 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 54 of its 72 references that are in the map. First posted on arXiv 22 Apr 2025.

Builds on

54 of 72 references.

Closest ancestors · 14

Same direction, newest first

  1. 2025Right Question is Already Half the Answer: Fully Unsupervised LLM Reasoning Incentivization8 Apr
  2. 2025GenPRM: Scaling Test-Time Compute of Process Reward Models via Generative Reasoning1 Apr
  3. 2025Open-Reasoner-Zero: An Open Source Approach to Scaling Up Reinforcement Learning on the Base Model31 Mar
  4. 2025Understanding R1-Zero-Like Training: A Critical Perspective26 Mar
  5. 2025SimpleRL-Zoo: Investigating and Taming Zero Reinforcement Learning for Open Base Models in the Wild24 Mar
  6. 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scale18 Mar
  7. 2025SWE-RL: Advancing LLM Reasoning via Reinforcement Learning on Open Software Evolution25 Feb
  8. 2025LIMR: Less is More for RL Scaling17 Feb
  9. 2025Process Reinforcement through Implicit Rewards3 Feb
  10. 2024Free Process Rewards without Process Labels2 Dec
  11. 2023Math-Shepherd: Verify and Reinforce LLMs Step-by-step without Human Annotations14 Dec
  12. 2023Let's Verify Step by Step31 May
2 more
  1. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Model29 May
  2. 2017Proximal Policy Optimization Algorithms20 Jul

Other references · 40

  1. 2022Training language models to follow instructions with human feedbackcited by 15,971
  2. 2024The Llama 3 Herd of Modelscited by 14,715
  3. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  4. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  5. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  6. 2020Scaling Laws for Neural Language Modelscited by 6,586
  7. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,708
  8. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,371
  9. 2024Qwen2.5 Technical Reportcited by 4,942
  10. 2023Self-Refine: Iterative Refinement with Self-Feedbackcited by 3,442
  11. 2023GPQA: A Graduate-Level Google-Proof Q&A Benchmarkcited by 2,879
  12. 2020Learning to summarize from human feedbackcited by 2,588
28 more
  1. 2024Qwen2 Technical Reportcited by 2,120
  2. 2024Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameterscited by 1,986
  3. 2021WebGPT: Browser-assisted question-answering with human feedbackcited by 1,745
  4. 2024Self-Rewarding Language Modelscited by 681
  5. 2025SFT Memorizes, RL Generalizes: A Comparative Study of Foundation Model Post-trainingcited by 672
  6. 2022Large Language Models Can Self-Improvecited by 664
  7. 2025BrowseComp: A Simple Yet Challenging Benchmark for Browsing Agentscited by 647
  8. 2023Language Agent Tree Search Unifies Reasoning Acting and Planning in Language Modelscited by 574
  9. 2024Self-Play Fine-Tuning Converts Weak Language Models to Strong Language Modelscited by 557
  10. 2023A Comprehensive Survey on Test-Time Adaptation under Distribution Shiftscited by 359
  11. 2024Learning to (Learn at Test Time): RNNs with Expressive Hidden Statescited by 355
  12. 2024Titans: Learning to Memorize at Test Timecited by 328
  13. 2024Monte Carlo Tree Search Boosts Reasoning via Iterative Preference Learningcited by 241
  14. 2023Universal Self-Consistency for Large Language Model Generationcited by 203
  15. 2024From Decoding to Meta-Generation: Inference-time Algorithms for Large Language Modelscited by 137
  16. 2025Can 1B LLM Surpass 405B LLM? Rethinking Compute-Optimal Test-Time Scalingcited by 136
  17. 2025Competitive Programming with Large Reasoning Modelscited by 119
  18. 2024ARGS: Alignment as Reward-Guided Searchcited by 103
  19. 2025One-Minute Video Generation with Test-Time Trainingcited by 101
  20. 2023Test-Time Training on Nearest Neighbors for Large Language Modelscited by 88
  21. 2018Reinforcement Learning with Perturbed Rewardscited by 79
  22. 2023Reward-Augmented Decoding: Efficient Controlled Text Generation With a Unidirectional Reward Modelcited by 79
  23. 2025What Makes a Reward Model a Good Teacher? An Optimization Perspectivecited by 73
  24. 2019Test-Time Training with Self-Supervision for Generalization under Distribution Shiftscited by 59
  25. 2024Self-Consistency Preference Optimizationcited by 50
  26. 2025Inference-Time Scaling for Complex Tasks: Where We Stand and What Lies Aheadcited by 44
  27. 2023Test-Time Training on Video Streamscited by 37
  28. 2025Genius: A Generalizable and Purely Unsupervised Self-Training Framework For Advanced Reasoningcited by 24

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 126; 2026: 149.

275 papers.

Sort:

By area:

  1. 2025The Entropy Mechanism of Reinforcement Learning for Reasoning Language Modelscited by 485
  2. 2026Reinforcement Learning via Self-Distillationcited by 332
  3. 2025Reinforcement Learning for Reasoning in Large Language Models with One Training Examplecited by 295
  4. 2025Reasoning with Exploration: An Entropy Perspectivecited by 281
  5. 2025R-Zero: Self-Evolving Reasoning LLM from Zero Datacited by 228
  6. 2025Learning to Reason without External Rewardscited by 210
  7. 2025Spurious Rewards: Rethinking Training Signals in RLVRcited by 209
  8. 2025The Landscape of Agentic Reinforcement Learning for LLMs: A Surveycited by 205
  9. 2025A Survey of Reinforcement Learning for Large Reasoning Modelscited by 180
  10. 2025The Unreasonable Effectiveness of Entropy Minimization in LLM Reasoningcited by 166
  11. 2025Skywork Open Reasoner 1 Technical Reportcited by 165
  12. 2025SimpleVLA-RL: Scaling VLA Training via Reinforcement Learningcited by 151
38 more
  1. 2025Pass@k Training for Adaptively Balancing Exploration and Exploitation of Large Reasoning Modelscited by 138
  2. 2025Maximizing Confidence Alone Improves Reasoningcited by 100
  3. 2025Reasoning or Memorization? Unreliable Results of Reinforcement Learning Due to Data Contaminationcited by 82
  4. 2025Reinforcing General Reasoning without Verifierscited by 77
  5. 2025Agent0: Unleashing Self-Evolving Agents from Zero Data via Tool-Integrated Reasoningcited by 75
  6. 2025Can Large Reasoning Models Self-Train?cited by 68
  7. 2026Helios: Real Real-Time Long Video Generation Modelcited by 65
  8. 2025Breaking the Exploration Bottleneck: Rubric-Scaffolded Reinforcement Learning for General LLM Reasoningcited by 63
  9. 2025Confidence Is All You Need: Few-Shot RL Fine-Tuning of Language Modelscited by 62
  10. 2025Parallel-R1: Towards Parallel Thinking via Reinforcement Learningcited by 61
  11. 2025SeRL: Self-Play Reinforcement Learning for Large Language Models with Limited Datacited by 51
  12. 2025Self-Questioning Language Modelscited by 50
  13. 2026A Survey of Agentic Reasoning for Large Language Models: Towards Recursively Self-Improving and Collective Agentscited by 50
  14. 2025Information Gain-based Policy Optimization: A Simple and Effective Approach for Multi-Turn Search Agentscited by 48
  15. 2025Consistent Paths Lead to Truth: Self-Rewarding Reinforcement Learning for LLM Reasoningcited by 46
  16. 2025Harnessing Uncertainty: Entropy-Modulated Policy Gradients for Long-Horizon LLM Agentscited by 46
  17. 2025Evolving Language Models without Labels: Majority Drives Selection, Novelty Promotes Variationcited by 45
  18. 2025ZeroGUI: Automating Online GUI Learning at Zero Human Costcited by 44
  19. 2026Dr. Zero: Self-Evolving Search Agents without Training Datacited by 44
  20. 2025Diversity-Aware Policy Optimization for Large Language Model Reasoningcited by 41
  21. 2025No Free Lunch: Rethinking Internal Feedback for LLM Reasoningcited by 37
  22. 2025Jigsaw-R1: A Study of Rule-based Visual Reinforcement Learning with Jigsaw Puzzlescited by 36
  23. 2025ETTRL: Balancing Exploration and Exploitation in LLM Test-Time Reinforcement Learning Via Entropy Mechanismcited by 36
  24. 2025Towards a Unified View of Large Language Model Post-Trainingcited by 36
  25. 2025Guided Self-Evolving LLMs with Minimal Human Supervisioncited by 35
  26. 2025First SFT, Second RL, Third UPT: Continual Improving Multi-Modal LLM Reasoning via Unsupervised Post-Trainingcited by 34
  27. 2026Tool-R0: Self-Evolving LLM Agents for Tool-Learning from Zero Datacited by 34
  28. 2025UserRL: Training Interactive User-Centric Agent via Reinforcement Learningcited by 32
  29. 2025Agent0-VL: Exploring Self-Evolving Agent for Tool-Integrated Vision-Language Reasoningcited by 31
  30. 2025JarvisEvo: Towards a Self-Evolving Photo Editing Agent with Synergistic Editor-Evaluator Optimizationcited by 31
  31. 2025Self-Improving LLM Agents at Test-Timecited by 30
  32. 2025AlphaOne: Reasoning Models Thinking Slow and Fast at Test Timecited by 29
  33. 2025SPELL: Self-Play Reinforcement Learning for Evolving Long-Context Language Modelscited by 29
  34. 2025CoMAS: Co-Evolving Multi-Agent Systems via Interaction Rewardscited by 29
  35. 2025PixelThink: Towards Efficient Chain-of-Pixel Reasoningcited by 28
  36. 2025Nudging the Boundaries of LLM Reasoningcited by 26
  37. 2026FIPO: Eliciting Deep Reasoning with Future-KL Influenced Policy Optimizationcited by 25
  38. 2026Rethinking On-Policy Distillation of Large Language Models II: One Training Examplecited by 24

50 most cited of 275 All 275 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026SelectTTRL: Representation-Guided Data Selection for Efficient Test-Time Reinforcement Learning
  2. 2025JustRL: Scaling a 1.5B LLM with a Simple RL Recipe
  3. 2026TR-ICRL: Test-Time Rethinking for In-Context Reinforcement Learning
  4. 2025RoiRL: Efficient, Self-Supervised Reasoning with Offline Iterative Reinforcement Learning
  5. 2026Hi-TTRL: Regulating Consensus with Hints for Test-Time Reinforcement Learning
  6. 2026A Survey of Reinforcement Learning for Large Language Models under Data Scarcity: Challenges and Solutions
  7. 2025Step-level Verifier-guided Hybrid Test-Time Scaling for Large Language Models
  8. 2025ProRL: Prolonged Reinforcement Learning Expands Reasoning Boundaries in Large Language Models