# Continual Learning Bench: Evaluating Frontier AI Systems in Real-World Stateful Environments

arXiv: https://arxiv.org/abs/2606.05661 · HTML: https://mlsanity.com/paper/2606.05661/

## Key facts

- Cited by 18 papers in the map (data through 14 Oct 2026).
- Builds on 21 of its 33 references that are in the map.
- First posted on arXiv 4 Jun 2026.

## Builds on (most cited first, 21 of 21)

- [SWE-bench: Can Language Models Resolve Real-World GitHub Issues?](https://mlsanity.com/paper/2310.06770/) (2023)
- [τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains](https://mlsanity.com/paper/2406.12045/) (2024)
- [Evaluating Very Long-Term Conversational Memory of LLM Agents](https://mlsanity.com/paper/2402.17753/) (2024)
- [Mem0: Building Production-Ready AI Agents with Scalable Long-Term Memory](https://mlsanity.com/paper/2504.19413/) (2025)
- [BrowseComp: A Simple Yet Challenging Benchmark for Browsing Agents](https://mlsanity.com/paper/2504.12516/) (2025)
- [Agentic Context Engineering: Evolving Contexts for Self-Improving Language Models](https://mlsanity.com/paper/2510.04618/) (2025)
- [Learning to (Learn at Test Time): RNNs with Expressive Hidden States](https://mlsanity.com/paper/2407.04620/) (2024)
- [TheAgentCompany: Benchmarking LLM Agents on Consequential Real World Tasks](https://mlsanity.com/paper/2412.14161/) (2024)
- [Continual World: A Robotic Benchmark For Continual Reinforcement Learning](https://mlsanity.com/paper/2105.10919/) (2021)
- [Cartridges: Lightweight and general-purpose long context representations via self-study](https://mlsanity.com/paper/2506.06266/) (2025)
- [LifelongAgentBench: Evaluating LLM Agents as Lifelong Learners](https://mlsanity.com/paper/2505.11942/) (2025)
- [Finance Agent Benchmark: Benchmarking LLMs on Real-world Financial Research Tasks](https://mlsanity.com/paper/2508.00828/) (2025)
- [PostTrainBench: Can LLM Agents Automate LLM Post-Training?](https://mlsanity.com/paper/2603.08640/) (2026)
- [End-to-End Test-Time Training for Long Context](https://mlsanity.com/paper/2512.23675/) (2025)
- [MemoryBench: A Benchmark for Memory and Continual Learning in LLM Systems](https://mlsanity.com/paper/2510.17281/) (2025)
- [SkillLearnBench: Benchmarking Continual Learning Methods for Agent Skill Generation on Real-World Tasks](https://mlsanity.com/paper/2604.20087/) (2026)
- [Continual evaluation for lifelong learning: Identifying the stability gap](https://mlsanity.com/paper/2205.13452/) (2022)
- [LongHealth: A Question Answering Benchmark with Long Clinical Documents](https://mlsanity.com/paper/2401.14490/) (2024)
- [SlopCodeBench: Benchmarking How Coding Agents Degrade Over Long-Horizon Iterative Tasks](https://mlsanity.com/paper/2603.24755/) (2026)
- [SWE-Bench-CL: Continual Learning for Coding Agents](https://mlsanity.com/paper/2507.00014/) (2025)
- [SIEVE: Sample-Efficient Parametric Learning from Natural Language](https://mlsanity.com/paper/2604.02339/) (2026)

## Built on by (most cited first, 18 of 18)

- [VibeLifeBench: Can Your Life Agent Be Proactive and Persistent in a Living World?](https://mlsanity.com/paper/2608.10875/) (2026)
- [EvoHarnessBench: Can Your Agents Keep Pace with an Evolving Harness?](https://mlsanity.com/paper/2609.04280/) (2026)
- [PATH-Bench: Path-Dependent Evaluation of Lifelong Agents](https://mlsanity.com/paper/2608.01149/) (2026)
- [Ouroboros: A Self-Developing Frontier Coding Agent with Reviewed Core Evolution](https://mlsanity.com/paper/2608.08311/) (2026)
- [Self-Evolving Coding Agents](https://mlsanity.com/paper/2608.03392/) (2026)
- [Connect the Dots: Training LLMs for Long-Lifecycle Agents with Cross-Domain Generalization Via Reinforcement Learning](https://mlsanity.com/paper/2606.20002/) (2026)
- [Rethinking Transfer in Continual Learning: A Replay-Based Realisation](https://mlsanity.com/paper/2607.15587/) (2026)
- [ExplorationBench: Measuring AI Systems' Exploration in Verifiable Alien Worlds](https://mlsanity.com/paper/2609.30199/) (2026)
- [Measure Learning at Steady State: A BIRD-SQL Formula 1 Case Study](https://mlsanity.com/paper/2609.31640/) (2026)
- [Can Agents Design Libraries for Agents?](https://mlsanity.com/paper/2609.36730/) (2026)
- [VIGIL: Verifier-Informed Gated Improvement Loop for Spreadsheet Question Answering](https://mlsanity.com/paper/2610.04287/) (2026)
- [ServeLearnBench: How Well Can Agents Self-Improve from Serving Experience?](https://mlsanity.com/paper/2610.07792/) (2026)
- [Agent Plasticity: Measuring Self-Improvement Through Experience](https://mlsanity.com/paper/2610.08902/) (2026)
- [From a Prompt to Repertoires: Evolving Functional REpertoires Enable LLM Continual Learning](https://mlsanity.com/paper/2610.11373/) (2026)
- [Tracing the Thoughts of a Coding Agent Playing ARC-AGI-3: Lessons for Continual Learning](https://mlsanity.com/paper/2610.11450/) (2026)
- [Recurrent Harness for Agentic Continual Learning](https://mlsanity.com/paper/2610.12668/) (2026)
- [From Existing Methods to New Research Ideas: Two Perspectives on Recursive Self-Improvement](https://mlsanity.com/paper/2610.14625/) (2026)
- [Reliable Knowledge Acquisition for Continual Learning Agents](https://mlsanity.com/paper/2610.21739/) (2026)

## Abstract (arXiv)

Continual learning, the ability of AI systems to improve through sequential experience, has attracted substantial interest, but no high-quality benchmark exists to evaluate it. We introduce Continual Learning Bench (CL-Bench), the first difficult, expert-validated benchmark designed to measure whether LLM-based systems genuinely improve with experience. CL-Bench spans six diverse domains (software engineering, signal processing, disease outbreak forecasting, database querying, strategic game-playing, and demand forecasting), each validated by domain experts and designed so that tasks share a learnable latent structure (codebase layout, disease outbreak dynamics, opponent strategies) that a stateful system can discover online but a stateless one cannot. We evaluate frontier models across several agent architectures, from naive in-context learning (ICL) to dedicated memory systems, introducing a gain metric to isolate learning from prior capabilities. We find that these systems leave headroom for improved continual learning: agents frequently overfit to immediate observations or fail to reuse knowledge across instances, and dedicated memory systems do not fix this -- in fact, naive ICL outperforms systems dedicated to memory management. CL-Bench is the first benchmark to evaluate continual learning across diverse real-world domains with expert-validated tasks and isolate online learning from underlying model capability, showing a need for better continual learning systems.