# Continuous Self-Improvement of Large Language Models by Test-time Training with Verifier-Driven Sample Selection

arXiv: https://arxiv.org/abs/2505.19475 · HTML: https://mlsanity.com/paper/2505.19475/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- Placed in Synthetic reasoning data and bootstrapping, one of 13 directions in Synthetic data and self-improvement for LLMs.
- Builds on 21 of its 28 references that are in the map.
- First posted on arXiv 26 May 2025.

## Placed in

- Synthetic data and self-improvement for LLMs / [Synthetic reasoning data and bootstrapping](https://mlsanity.com/synth/synthetic-reasoning-data-and-bootstrapping/)

## Builds on (most cited first, 21 of 21)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Training Verifiers to Solve Math Word Problems](https://mlsanity.com/paper/2110.14168/) (2021)
- [DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models](https://mlsanity.com/paper/2402.03300/) (2024)
- [Measuring Mathematical Problem Solving With the MATH Dataset](https://mlsanity.com/paper/2103.03874/) (2021)
- [Let's Verify Step by Step](https://mlsanity.com/paper/2305.20050/) (2023)
- [Reinforced Self-Training (ReST) for Language Modeling](https://mlsanity.com/paper/2308.08998/) (2023)
- [Efficient Test-Time Model Adaptation without Forgetting](https://mlsanity.com/paper/2204.02610/) (2022)
- [Weak-to-Strong Generalization: Eliciting Strong Capabilities With Weak Supervision](https://mlsanity.com/paper/2312.09390/) (2023)
- [Open-Reasoner-Zero: An Open Source Approach to Scaling Up Reinforcement Learning on the Base Model](https://mlsanity.com/paper/2503.24290/) (2025)
- [ReST-MCTS*: LLM Self-Training via Process Reward Guided Tree Search](https://mlsanity.com/paper/2406.03816/) (2024)
- [TTRL: Test-Time Reinforcement Learning](https://mlsanity.com/paper/2504.16084/) (2025)
- [The Effect of Sampling Temperature on Problem Solving in Large Language Models](https://mlsanity.com/paper/2402.05201/) (2024)
- [Continual Learning for Large Language Models: A Survey](https://mlsanity.com/paper/2402.01364/) (2024)
- [Test-Time Training on Nearest Neighbors for Large Language Models](https://mlsanity.com/paper/2305.18466/) (2023)
- [Scaling Test-Time Compute Without Verification or RL is Suboptimal](https://mlsanity.com/paper/2502.12118/) (2025)
- [MT3: Meta Test-Time Training for Self-Supervised Test-Time Adaption](https://mlsanity.com/paper/2103.16201/) (2021)
- [Beyond Model Adaptation at Test Time: A Survey](https://mlsanity.com/paper/2411.03687/) (2024)
- [Efficiently Learning at Test-Time: Active Fine-Tuning of LLMs](https://mlsanity.com/paper/2410.08020/) (2024)
- [Continual Test-time Domain Adaptation via Dynamic Sample Selection](https://mlsanity.com/paper/2310.03335/) (2023)
- [The Entropy Enigma: Success and Failure of Entropy Minimization](https://mlsanity.com/paper/2405.05012/) (2024)
- [SuperHF: Supervised Iterative Learning from Human Feedback](https://mlsanity.com/paper/2310.16763/) (2023)

## Built on by (most cited first, 13 of 13)

- [RL's Razor: Why Online Reinforcement Learning Forgets Less](https://mlsanity.com/paper/2509.04259/) (2025)
- [Why AI systems don't learn and what to do about it: Lessons on autonomous learning from cognitive science](https://mlsanity.com/paper/2603.15381/) (2026)
- [Locas: Your Models are Principled Initializers of Locally-Supported Parametric Memories](https://mlsanity.com/paper/2602.05085/) (2026)
- [Gated Differentiable Working Memory for Long-Context Language Modeling](https://mlsanity.com/paper/2601.12906/) (2026)
- [Scaling Large Reasoning Models beyond Human Supervision: A Path toward Superintelligence](https://mlsanity.com/paper/2608.31075/) (2026)
- [DiSCTT: Consensus-Guided Self-Curriculum for Efficient Test-Time Adaptation in Reasoning](https://mlsanity.com/paper/2603.05357/) (2026)
- [Training LLM Agents for Spontaneous, Reward-Free Self-Evolution via World Knowledge Exploration](https://mlsanity.com/paper/2604.18131/) (2026)
- [Phantom Gains: Auditing Self-Improvement Against a Measured Null](https://mlsanity.com/paper/2608.20290/) (2026)
- [ETT: Expanding the Long Context Understanding Capability of LLMs at Test-Time](https://mlsanity.com/paper/2507.06313/) (2025)
- [FLOP-Efficient Training: Early Stopping Based on Test-Time Compute Awareness](https://mlsanity.com/paper/2601.01332/) (2026)
- [Epistemic Uncertainty for Test-Time Discovery](https://mlsanity.com/paper/2605.11328/) (2026)
- [SOLAR: A Self-Optimizing Open-Ended Autonomous Agent for Lifelong Learning and Continual Adaptation](https://mlsanity.com/paper/2605.20189/) (2026)
- [Rewarding Reasoning, Not Answers: Fixing and Bounding Test-Time Reinforcement Learning on Medical QA](https://mlsanity.com/paper/2609.16660/) (2026)

## Abstract (arXiv)

Learning to adapt pretrained language models to unlabeled, out-of-distribution data is a critical challenge, as models often falter on structurally novel reasoning tasks even while excelling within their training distribution. We introduce a new framework called VDS-TTT - Verifier-Driven Sample Selection for Test-Time Training to efficiently address this. We use a learned verifier to score a pool of generated responses and select only from high ranking pseudo-labeled examples for fine-tuned adaptation. Specifically, for each input query our LLM generates N candidate answers; the verifier assigns a reliability score to each, and the response with the highest confidence and above a fixed threshold is paired with its query for test-time training. We fine-tune only low-rank LoRA adapter parameters, ensuring adaptation efficiency and fast convergence. Our proposed self-supervised framework is the first to synthesize verifier driven test-time training data for continuous self-improvement of the model. Experiments across three diverse benchmarks and three state-of-the-art LLMs demonstrate that VDS-TTT yields up to a 32.29% relative improvement over the base model and a 6.66% gain compared to verifier-based methods without test-time training, highlighting its effectiveness and efficiency for on-the-fly large language model adaptation.