# EvoHarnessBench: Can Your Agents Keep Pace with an Evolving Harness? arXiv: https://arxiv.org/abs/2609.04280 · HTML: https://mlsanity.com/paper/2609.04280/ ## Key facts - Cited by 4 papers in the map (data through 14 Oct 2026). - Placed in Software engineering coding agents, one of 16 directions in LLM agents and tool use. - Builds on 33 of its 40 references that are in the map. - First posted on arXiv 3 Sep 2026. ## Placed in - LLM agents and tool use / [Software engineering coding agents](https://mlsanity.com/agents/software-engineering-coding-agents/) ## Builds on (most cited first, 30 of 33) - [ReAct: Synergizing Reasoning and Acting in Language Models](https://mlsanity.com/paper/2210.03629/) (2022) - [AutoGen: Enabling Next-Gen LLM Applications via Multi-Agent Conversation](https://mlsanity.com/paper/2308.08155/) (2023) - [τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains](https://mlsanity.com/paper/2406.12045/) (2024) - [A Comprehensive Survey of Continual Learning: Theory, Method and Application](https://mlsanity.com/paper/2302.00487/) (2023) - [API-Bank: A Comprehensive Benchmark for Tool-Augmented LLMs](https://mlsanity.com/paper/2304.08244/) (2023) - [GEPA: Reflective Prompt Evolution Can Outperform Reinforcement Learning](https://mlsanity.com/paper/2507.19457/) (2025) - [AFlow: Automating Agentic Workflow Generation](https://mlsanity.com/paper/2410.10762/) (2024) - [Automated Design of Agentic Systems](https://mlsanity.com/paper/2408.08435/) (2024) - [PaperBench: Evaluating AI's Ability to Replicate AI Research](https://mlsanity.com/paper/2504.01848/) (2025) - [SkillsBench: Benchmarking How Well Agent Skills Work Across Diverse Tasks](https://mlsanity.com/paper/2602.12670/) (2026) - [Meta-Harness: End-to-End Optimization of Model Harnesses](https://mlsanity.com/paper/2603.28052/) (2026) - [ReasoningBank: Scaling Agent Self-Evolving with Reasoning Memory](https://mlsanity.com/paper/2509.25140/) (2025) - [KernelBench: Can LLMs Write Efficient GPU Kernels?](https://mlsanity.com/paper/2502.10517/) (2025) - [RE-Bench: Evaluating frontier AI R&D capabilities of language model agents against human experts](https://mlsanity.com/paper/2411.15114/) (2024) - [CORE-Bench: Fostering the Credibility of Published Research Through a Computational Reproducibility Agent Benchmark](https://mlsanity.com/paper/2409.11363/) (2024) - [G-Memory: Tracing Hierarchical Memory for Multi-Agent Systems](https://mlsanity.com/paper/2506.07398/) (2025) - [A Survey of Frontiers in LLM Reasoning: Inference Scaling, Learning to Reason, and Agentic Systems](https://mlsanity.com/paper/2504.09037/) (2025) - [Externalization in LLM Agents: A Unified Review of Memory, Skills, Protocols and Harness Engineering](https://mlsanity.com/paper/2604.08224/) (2026) - [How Well Do Agentic Skills Work in the Wild: Benchmarking LLM Skill Usage in Realistic Settings](https://mlsanity.com/paper/2604.04323/) (2026) - [SkillLearnBench: Benchmarking Continual Learning Methods for Agent Skill Generation on Real-World Tasks](https://mlsanity.com/paper/2604.20087/) (2026) - [LEGOMem: Modular Procedural Memory for Multi-agent LLM Systems for Workflow Automation](https://mlsanity.com/paper/2510.04851/) (2025) - [MAS-Orchestra: Understanding and Improving Multi-Agent Reasoning Through Holistic Orchestration and Controlled Benchmarks](https://mlsanity.com/paper/2601.14652/) (2026) - [Agents' Last Exam](https://mlsanity.com/paper/2606.05405/) (2026) - [MAS-ZERO: Designing Multi-Agent Systems with Zero Supervision](https://mlsanity.com/paper/2505.14996/) (2025) - [Continual Learning Bench: Evaluating Frontier AI Systems in Real-World Stateful Environments](https://mlsanity.com/paper/2606.05661/) (2026) - [EnterpriseOps-Gym: Environments and Evaluations for Stateful Agentic Planning and Tool Use in Enterprise Settings](https://mlsanity.com/paper/2603.13594/) (2026) - [Harness Engineering for Agentic AI Coding Tools: An Exploratory Study](https://mlsanity.com/paper/2602.14690/) (2026) - [Decentralized Multi-Agent Systems with Shared Context](https://mlsanity.com/paper/2606.10662/) (2026) - [AgentCL: Toward Rigorous Evaluation of Continual Learning in Language Agents](https://mlsanity.com/paper/2606.02461/) (2026) - [From Model Scaling to System Scaling: Scaling the Harness in Agentic AI](https://mlsanity.com/paper/2605.26112/) (2026) ## Built on by (most cited first, 4 of 4) - [RRSI: Regularized Recursive Self-Improvement of Agent Harnesses](https://mlsanity.com/paper/2609.24972/) (2026) - [Cortex: Measurement-Gated Evolution and Pruning of Coding-Agent Skills and Rules](https://mlsanity.com/paper/2610.14216/) (2026) - [Why Do Agent Teams Fail Across Boundaries? A Failure Taxonomy for Agent-to-Agent Systems with Private States](https://mlsanity.com/paper/2610.19342/) (2026) - [Doc2Harness: Benchmarking LLMs on Long-Horizon Harness Engineering of Coding Agents](https://mlsanity.com/paper/2610.20344/) (2026) ## Abstract (arXiv) Modern LLM-based agents operate through a harness of tools, reusable skills, and specialist agents that shapes what they observe and what they can do. In practice, this harness continually evolves as new capabilities are added. We introduce EvoHarnessBench, a benchmark for evaluating agents under controlled harness evolution across three axes (tools, skills, and agents). Unlike existing continual-learning benchmarks for agents, which typically place non-stationarity (i.e., what changes over time) in the task stream while keeping the harness fixed, EvoHarnessBench places non-stationarity in the externally supplied harness itself. It contains 17 multi-stage harness streams constructed deterministically from verifier-based benchmarks, comprising 802 tasks, 520 tools, 42 skills, and 62 agents. We evaluate two complementary settings corresponding to the central challenges of harness evolution: deployment evaluation, which isolates retention of previously accessible competence as the harness expands, and self-evolving adaptation evaluation, which tests whether accumulated experience remains useful as new capabilities are introduced. Our results reveal three persistent gaps. First, harness expansion alone can degrade performance on previously solved tasks, producing harness-induced forgetting. Second, gains from self-evolving adaptation remain inconsistent across stages of harness evolution, capability axes, and environments. Third, retention and adaptation can pull in different directions: preserving earlier competence does not necessarily improve adaptation to newly introduced capabilities, and vice versa. These results establish harness evolution as a distinct challenge for building agents that can keep pace with an evolving harness while preserving previously effective behavior.