# AEL: Evolving Agent Harness in Open-Ended Environments

arXiv: https://arxiv.org/abs/2604.21725 · HTML: https://mlsanity.com/paper/2604.21725/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Placed in Agent memory systems, one of 16 directions in LLM agents and tool use.
- Builds on 17 of its 29 references that are in the map.
- First posted on arXiv 23 Apr 2026.

## Placed in

- LLM agents and tool use / [Agent memory systems](https://mlsanity.com/agents/agent-memory-systems/)

## Builds on (most cited first, 17 of 17)

- [ReAct: Synergizing Reasoning and Acting in Language Models](https://mlsanity.com/paper/2210.03629/) (2022)
- [A Unified Approach to Interpreting Model Predictions](https://mlsanity.com/paper/1705.07874/) (2017)
- [Toolformer: Language Models Can Teach Themselves to Use Tools](https://mlsanity.com/paper/2302.04761/) (2023)
- [SWE-bench: Can Language Models Resolve Real-World GitHub Issues?](https://mlsanity.com/paper/2310.06770/) (2023)
- [WebArena: A Realistic Web Environment for Building Autonomous Agents](https://mlsanity.com/paper/2307.13854/) (2023)
- [MemGPT: Towards LLMs as Operating Systems](https://mlsanity.com/paper/2310.08560/) (2023)
- [A-MEM: Agentic Memory for LLM Agents](https://mlsanity.com/paper/2502.12110/) (2025)
- [ExpeL: LLM Agents Are Experiential Learners](https://mlsanity.com/paper/2308.10144/) (2023)
- [Understanding the planning of LLM agents: A survey](https://mlsanity.com/paper/2402.02716/) (2024)
- [Mem-Gallery: Benchmarking Multimodal Long-Term Conversational Memory for MLLM Agents](https://mlsanity.com/paper/2601.03515/) (2026)
- [EvoTool: Self-Evolving Tool-Use Policy Optimization in LLM Agents via Blame-Aware Mutation and Diversity-Aware Selection](https://mlsanity.com/paper/2603.04900/) (2026)
- [SLMRec: Distilling Large Language Models into Small for Sequential Recommendation](https://mlsanity.com/paper/2405.17890/) (2024)
- [EPO: Entropy-regularized Policy Optimization for LLM Agents Reinforcement Learning](https://mlsanity.com/paper/2509.22576/) (2025)
- [FactorMiner: A Self-Evolving Agent with Skills and Experience Memory for Financial Alpha Discovery](https://mlsanity.com/paper/2602.14670/) (2026)
- [Meta-Policy Reflexion: Reusable Reflective Memory and Rule Admissibility for Resource-Efficient LLM Agent](https://mlsanity.com/paper/2509.03990/) (2025)
- [AutoAgent: Evolving Cognition and Elastic Memory Orchestration for Adaptive Agents](https://mlsanity.com/paper/2603.09716/) (2026)
- [Tool-Genesis: A Task-Driven Tool Creation Benchmark for Self-Evolving Language Agent](https://mlsanity.com/paper/2603.05578/) (2026)

## Built on by (most cited first, 6 of 6)

- [MemEye: A Visual-Centric Evaluation Framework for Multimodal Agent Memory](https://mlsanity.com/paper/2605.15128/) (2026)
- [Co-Evolving Skill Generation and Policy Optimization](https://mlsanity.com/paper/2606.08755/) (2026)
- [Emotion2Skill: Model-Internal Emotion Signals for Adaptive Skill Selection and Evolution](https://mlsanity.com/paper/2608.09248/) (2026)
- [How do Humans Process AI-generated Hallucination Contents: a Neuroimaging Study](https://mlsanity.com/paper/2605.16953/) (2026)
- [OpenPM: Auditable Point-in-Time Evaluation for LLM Portfolio-Management Agents](https://mlsanity.com/paper/2608.09988/) (2026)
- [S3: Spectral Null-Space Swap Makes Reasoning Models Efficient](https://mlsanity.com/paper/2609.37976/) (2026)

## Abstract (arXiv)

LLM Agents Harnesses are hand-designed and stay fixed, so agents accumulate experience but never learn how to use it: which memories to retrieve, when retrieved evidence is misleading, and when the retrieval strategy itself should change. We introduce Agent Evolving Learning (AEL), a two-timescale framework that evolves the harness, recasting memory use as online policy selection. A fast Thompson Sampling bandit selects among memory-retrieval policies episode by episode, while slow LLM reflection follows a diagnose-before-prescribe principle: it first explains why performance degraded, then injects a targeted new retrieval policy as a bandit arm when the current pool plateaus. AEL outperforms ten self-improving and nonLLM baselines on a sequential portfolio benchmark, lifting Sharpe by 27% over the strongest memory-only variant with the lowest variance among all stochastic methods, and generalizes to a support-ticket routing stream, where it improves accuracy by 18% over reflectionfree Thompson Sampling and by 51% over the best prior baseline. Mechanism studies further show that the gains are causal: reflection helps precisely when regimes demand different retrieval behavior, and is provably no-harm/nogain when the best policy is stable. Code and data: https://github.com/WujiangXu/AEL.