# Meta-TTL: Meta-Learning Self-Improvement Policies for Language Agents

arXiv: https://arxiv.org/abs/2604.00830 · HTML: https://mlsanity.com/paper/2604.00830/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 31 of its 45 references that are in the map.
- First posted on arXiv 1 Apr 2026.

## Builds on (most cited first, 30 of 31)

- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models](https://mlsanity.com/paper/2402.03300/) (2024)
- [Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks](https://mlsanity.com/paper/1703.03400/) (2017)
- [Reflexion: Language Agents with Verbal Reinforcement Learning](https://mlsanity.com/paper/2303.11366/) (2023)
- [On First-Order Meta-Learning Algorithms](https://mlsanity.com/paper/1803.02999/) (2018)
- [Meta-Learning in Neural Networks: A Survey](https://mlsanity.com/paper/2004.05439/) (2020)
- [STaR: Bootstrapping Reasoning With Reasoning](https://mlsanity.com/paper/2203.14465/) (2022)
- [Large Language Models](https://mlsanity.com/paper/2307.05782/) (2023)
- [Mem0: Building Production-Ready AI Agents with Scalable Long-Term Memory](https://mlsanity.com/paper/2504.19413/) (2025)
- [τ2-Bench: Evaluating Conversational Agents in a Dual-Control Environment](https://mlsanity.com/paper/2506.07982/) (2025)
- [Promptbreeder: Self-Referential Self-Improvement Via Prompt Evolution](https://mlsanity.com/paper/2309.16797/) (2023)
- [GEPA: Reflective Prompt Evolution Can Outperform Reinforcement Learning](https://mlsanity.com/paper/2507.19457/) (2025)
- [Training Language Models to Self-Correct via Reinforcement Learning](https://mlsanity.com/paper/2409.12917/) (2024)
- [Agentic Context Engineering: Evolving Contexts for Self-Improving Language Models](https://mlsanity.com/paper/2510.04618/) (2025)
- [ReEvo: Large Language Models as Hyper-Heuristics with Reflective Evolution](https://mlsanity.com/paper/2402.01145/) (2024)
- [TTRL: Test-Time Reinforcement Learning](https://mlsanity.com/paper/2504.16084/) (2025)
- [EvoPrompt: Connecting LLMs with Evolutionary Algorithms Yields Powerful Prompt Optimizers](https://mlsanity.com/paper/2309.08532/) (2023)
- [A Comprehensive Survey of Self-Evolving AI Agents: A New Paradigm Bridging Foundation Models and Lifelong Agentic Systems](https://mlsanity.com/paper/2508.07407/) (2025)
- [A Survey of Self-Evolving Agents: What, When, How, and Where to Evolve on the Path to Artificial Super Intelligence](https://mlsanity.com/paper/2507.21046/) (2025)
- [Interactive Fiction Games: A Colossal Adventure](https://mlsanity.com/paper/1909.05398/) (2019)
- [Evo-Memory: Benchmarking LLM Agent Test-time Learning with Self-Evolving Memory](https://mlsanity.com/paper/2511.20857/) (2025)
- [Learning to Discover at Test Time](https://mlsanity.com/paper/2601.16175/) (2026)
- [The Surprising Effectiveness of Test-Time Training for Few-Shot Learning](https://mlsanity.com/paper/2411.07279/) (2024)
- [Self-Adapting Language Models](https://mlsanity.com/paper/2506.10943/) (2025)
- [StreamBench: Towards Benchmarking Continuous Improvement of Language Agents](https://mlsanity.com/paper/2406.08747/) (2024)
- [Learning without training: The implicit dynamics of in-context learning](https://mlsanity.com/paper/2507.16003/) (2025)
- [EvoTest: Evolutionary Test-Time Learning for Self-Improving Agentic Systems](https://mlsanity.com/paper/2510.13220/) (2025)
- [Self-Improving LLM Agents at Test-Time](https://mlsanity.com/paper/2510.07841/) (2025)
- [Meta-RL Induces Exploration in Language Agents](https://mlsanity.com/paper/2512.16848/) (2025)
- [MetaReflection: Learning Instructions for Language Agents using Past Reflections](https://mlsanity.com/paper/2405.13009/) (2024)

## Built on by (most cited first, 3 of 3)

- [Environment-Grounded Automated Prompt Optimization for LLM Game Agents](https://mlsanity.com/paper/2606.17838/) (2026)
- [Experience-Sensitive Game Learning: A Behavioral Study of Humans and Language Agents](https://mlsanity.com/paper/2608.07490/) (2026)
- [Learn2Play Bench: How Well Do LLM Agents Learn from Experience in Unfamiliar Environments?](https://mlsanity.com/paper/2610.08215/) (2026)

## Abstract (arXiv)

Test-Time Learning (TTL) enables language agents to iteratively refine their performance through repeated interactions with the environment at inference time. At the core of TTL is a self-improvement policy that updates the actor policy based on experience from previous episodes, thereby improving future behavior. Existing methods rely on hand-crafted self-improvement rather than optimizing them for downstream improvement. We argue that optimal self-improvement policies should be learned from task environments, not hand-engineered based on human intuition. To achieve this, we introduce Meta-TTL, a framework that formulates the discovery of effective self-improvement policies as a bi-level optimization problem. Within this framework, the inner loop executes the standard TTL process, measuring how effectively a candidate self-improvement policy helps an agent correct errors across sequential episodes. Guided by the agent's performance, the outer loop performs reflective meta-training across diverse training tasks, using a balanced improvement score (BIS) to balance task contributions during candidate selection. We evaluate Meta-TTL on Jericho, WebArena-Lite, and -bench across both in-distribution (ID) and out-of-distribution (OOD) settings. Meta-TTL consistently outperforms existing baselines, improving TTL over the strongest baseline by up to 23% on ID tasks and 27% on OOD tasks. These results suggest that the optimized self-improvement policy encodes transferable meta-strategies that generalize beyond the training task distribution.