# SWE-Bench-CL: Continual Learning for Coding Agents

arXiv: https://arxiv.org/abs/2507.00014 · HTML: https://mlsanity.com/paper/2507.00014/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- Placed in Agent memory systems, one of 16 directions in LLM agents and tool use.
- Builds on 16 of its 22 references that are in the map.
- First posted on arXiv 13 Jun 2025.

## Placed in

- LLM agents and tool use / [Agent memory systems](https://mlsanity.com/agents/agent-memory-systems/)

## Builds on (most cited first, 16 of 16)

- [Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks](https://mlsanity.com/paper/2005.11401/) (2020)
- [Evaluating Large Language Models Trained on Code](https://mlsanity.com/paper/2107.03374/) (2021)
- [Program Synthesis with Large Language Models](https://mlsanity.com/paper/2108.07732/) (2021)
- [SWE-bench: Can Language Models Resolve Real-World GitHub Issues?](https://mlsanity.com/paper/2310.06770/) (2023)
- [Code Llama: Open Foundation Models for Code](https://mlsanity.com/paper/2308.12950/) (2023)
- [Gradient Episodic Memory for Continual Learning](https://mlsanity.com/paper/1706.08840/) (2017)
- [SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering](https://mlsanity.com/paper/2405.15793/) (2024)
- [Continual Lifelong Learning with Neural Networks: A Review](https://mlsanity.com/paper/1802.07569/) (2018)
- [CodeGen: An Open Large Language Model for Code with Multi-Turn Program Synthesis](https://mlsanity.com/paper/2203.13474/) (2022)
- [CodeXGLUE: A Machine Learning Benchmark Dataset for Code Understanding and Generation](https://mlsanity.com/paper/2102.04664/) (2021)
- [Riemannian Walk for Incremental Learning: Understanding Forgetting and Intransigence](https://mlsanity.com/paper/1801.10112/) (2018)
- [CodeSearchNet Challenge: Evaluating the State of Semantic Code Search](https://mlsanity.com/paper/1909.09436/) (2019)
- [Nomic Embed: Training a Reproducible Long Context Text Embedder](https://mlsanity.com/paper/2402.01613/) (2024)
- [Efficient Reinforcement Finetuning via Adaptive Curriculum Learning](https://mlsanity.com/paper/2504.05520/) (2025)
- [Emergent mechanisms for long timescales depend on training curriculum and affect performance in memory tasks](https://mlsanity.com/paper/2309.12927/) (2023)
- [Generalization of Medical Large Language Models through Cross-Domain Weak Supervision](https://mlsanity.com/paper/2502.00832/) (2025)

## Built on by (most cited first, 14 of 14)

- [SWE-CI: Evaluating Agent Capabilities in Maintaining Codebases via Continuous Integration](https://mlsanity.com/paper/2603.03823/) (2026)
- [Continual Learning Bench: Evaluating Frontier AI Systems in Real-World Stateful Environments](https://mlsanity.com/paper/2606.05661/) (2026)
- [MEMTRACK: Evaluating Long-Term Memory and State Tracking in Multi-Platform Dynamic Agent Environments](https://mlsanity.com/paper/2510.01353/) (2025)
- [Structurally Aligned Subtask-Level Memory for Software Engineering Agents](https://mlsanity.com/paper/2602.21611/) (2026)
- [AgentCL: Toward Rigorous Evaluation of Continual Learning in Language Agents](https://mlsanity.com/paper/2606.02461/) (2026)
- [Agentic Software Issue Resolution with Large Language Models: A Survey](https://mlsanity.com/paper/2512.22256/) (2025)
- [Agentic AI in the Software Development Lifecycle: Architecture, Empirical Evidence, and the Reshaping of Software Engineering](https://mlsanity.com/paper/2604.26275/) (2026)
- [Do Self-Evolving Agents Forget? Capability Degradation and Preservation in Lifelong LLM Agent Adaptation](https://mlsanity.com/paper/2605.09315/) (2026)
- [LatentGym: A Testbed For Cross-Task Experiential Learning With Controllable Latent Structure](https://mlsanity.com/paper/2606.15306/) (2026)
- [Is Agent Code Less Maintainable Than Human Code?](https://mlsanity.com/paper/2606.21804/) (2026)
- [MEGA: Self-Evolving Agent Optimization Infrastructure via Wisdom Graph](https://mlsanity.com/paper/2608.10504/) (2026)
- [How Do AI Coding Agents Contribute to Software Development? an Empirical Study of Agentic Pull Requests](https://mlsanity.com/paper/2607.21832/) (2026)
- [ExplainBench: Evaluating Code Explanations from Agents](https://mlsanity.com/paper/2607.26451/) (2026)
- [Learn2Play Bench: How Well Do LLM Agents Learn from Experience in Unfamiliar Environments?](https://mlsanity.com/paper/2610.08215/) (2026)

## Abstract (arXiv)

Large Language Models (LLMs) have achieved impressive results on static code-generation benchmarks, but real-world software development unfolds as a continuous stream of evolving issues, fixes, and feature requests. We introduce SWE-Bench-CL, a novel continual learning benchmark built on the human-verified SWE-Bench Verified dataset introduced by OpenAI and Princeton-NLP in 2024. By organizing GitHub issues into chronologically ordered sequences that reflect natural repository evolution, SWE-Bench-CL enables direct evaluation of an agent's ability to accumulate experience, transfer knowledge across tasks, and resist catastrophic forgetting. We complement the dataset with (i) a preliminary analysis of inter-task structural similarity and contextual sensitivity, (ii) an interactive LangGraph-based evaluation framework augmented with a FAISS-backed semantic memory module, and (iii) a suite of specialized continual learning metrics -- including average accuracy, forgetting, forward/backward transfer, tool-use efficiency, and a generalized Composite Continual Learning Score and CL-F-beta score -- to capture the stability-plasticity trade-off. We outline a rigorous experimental protocol comparing memory-enabled and memory-disabled agents across diverse Python repositories. All code and data are publicly available at https://github.com/thomasjoshi/agents-never-forget, providing the community with a reproducible platform for developing more adaptive and robust AI agents in software engineering.