mlsanityData to 13 Oct 2026

Harness Updating Is Not Harness Benefit: Disentangling Evolution Capabilities in Self-Evolving LLM Agents

arXiv 2605.30621 ↗First posted 28 May 2026cs.AIcited by 29

On the map: LLM agents and tool use → Software engineering coding agents

also in 1 more
Abstract (arXiv)

LLM agents are increasingly deployed as systems built around editable external harnesses, including prompts, skills, memories and tools, that shape task execution without changing model parameters. Harness self-evolution adapts such agents by updating these harnesses from execution evidence. Yet it remains unclear whether a model's base capability in task-solving predicts its capabilities in harness self-evolution: which models produce useful harness updates, and which actually benefit from them? We analyze two harness self-evolution capabilities: (i) harness-updating, the capability to produce useful persistent harness updates from execution evidence; (ii) harness-benefit, the capability to benefit from updated harnesses during task solving. Our analysis reveals two findings. First, harness-updating is flat in base capability: models from different capability tiers produce harness updates that lead to surprisingly similar gains; even Qwen3.5-9B's updates yield gains comparable to those of Claude Opus~4.6. Second, harness-benefit is non-monotonic in base capability: weak-tier models benefit little from updated harnesses, mid-tier models benefit most, and strong-tier models benefit less than mid-tier. We trace low gains at the weak tier to two failure modes: weak-tier models may fail to activate relevant harness artifacts, or activate them but fail to follow them faithfully. These findings suggest investing capability budget in the task-solving agent rather than the evolver, and targeting harness invocation and long-horizon instruction following in agent training. Our source code is publicly available at https://github.com/A-EVO-Lab/a-evolve/tree/release/harness-evolution.

Key facts. Cited by 29 papers in the map (data through 13 Oct 2026). Placed in Software engineering coding agents, one of 16 directions in LLM agents and tool use. Builds on 36 of its 48 references that are in the map. First posted on arXiv 28 May 2026.

Builds on

36 of 48 references.

Closest ancestors · 1

Same direction, newest first

  1. 2026Code as Agent Harness18 May

Other references · 35

  1. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,072
  2. 2025Qwen3 Technical Reportcited by 8,510
  3. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,502
  4. 2020Measuring Massive Multitask Language Understandingcited by 7,876
  5. 2023Self-Refine: Iterative Refinement with Self-Feedbackcited by 3,367
  6. 2023SWE-bench: Can Language Models Resolve Real-World GitHub Issues?cited by 3,274
  7. 2023ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIscited by 1,904
  8. 2025A-MEM: Agentic Memory for LLM Agentscited by 1,032
  9. 2022Large Language Models Are Human-Level Prompt Engineerscited by 1,031
  10. 2023ExpeL: LLM Agents Are Experiential Learnerscited by 842
  11. 2026Terminal-Bench: Benchmarking Agents on Hard, Realistic Tasks in Command Line Interfacescited by 547
  12. 2025GEPA: Reflective Prompt Evolution Can Outperform Reinforcement Learningcited by 489
23 more
  1. 2025Agentic Context Engineering: Evolving Contexts for Self-Improving Language Modelscited by 371
  2. 2026SkillsBench: Benchmarking How Well Agent Skills Work Across Diverse Taskscited by 302
  3. 2026Meta-Harness: End-to-End Optimization of Model Harnessescited by 278
  4. 2025ReasoningBank: Scaling Agent Self-Evolving with Reasoning Memorycited by 269
  5. 2026SkillRL: Evolving Agents via Recursive Skill-Augmented Reinforcement Learningcited by 268
  6. 2024ToolACE: Winning the Points of LLM Function Callingcited by 208
  7. 2025LightMem: Lightweight and Efficient Memory-Augmented Generationcited by 172
  8. 2025EvolveR: Self-Evolving LLM Agents through an Experience-Driven Lifecyclecited by 134
  9. 2025Evo-Memory: Benchmarking LLM Agent Test-time Learning with Self-Evolving Memorycited by 134
  10. 2026AutoSkill: Experience-Driven Lifelong Learning via Skill Self-Evolutioncited by 111
  11. 2025MemEvolve: Meta-Evolution of Agent Memory Systemscited by 84
  12. 2026Externalization in LLM Agents: A Unified Review of Memory, Skills, Protocols and Harness Engineeringcited by 73
  13. 2026Natural-Language Agent Harnessescited by 58
  14. 2026Graph-of-Skills: Dependency-Aware Structural Retrieval for Massive Agent Skillscited by 55
  15. 2026MCP-Atlas: A Large-Scale Benchmark for Tool-Use Competency with Real MCP Serverscited by 47
  16. 2024PromptWizard: Task-Aware Prompt Optimization Frameworkcited by 33
  17. 2025A Comprehensive Survey on Reinforcement Learning-based Agentic Search: Foundations, Roles, Optimizations, Evaluations, and Applicationscited by 22
  18. 2026Position: Agentic Evolution is the Path to Evolving LLMscited by 15
  19. 2026SEA-Eval: A Benchmark for Evaluating Self-Evolving Agents Beyond Episodic Assessmentcited by 13
  20. 2026MemMA: Coordinating the Memory Cycle through Multi-Agent Reasoning and In-Situ Self-Evolutioncited by 12
  21. 2025EngiBench: A Benchmark for Evaluating Large Language Models on Engineering Problem Solvingcited by 10
  22. 2026How Far Are LLMs from Professional Poker Players? Revisiting Game-Theoretic Reasoning with Agentic Tool Usecited by 7
  23. 2026Yunjue Agent Tech Report: A Fully Reproducible, Zero-Start In-Situ Self-Evolving Agent System for Open-Ended Taskscited by 4

Built on by

Cited by year (2026 to 13 Oct)

Citing papers per year: 2026: 29.

29 papers.

Sort:

By area:

  1. 2026Long-Horizon-Terminal-Bench: Testing the Limits of Agents on Long-Horizon Terminal Tasks with Dense Reward-Based Gradingcited by 22
  2. 2026HarnessDev: Can LLMs Create and Evolve Their Own Agent Harness?cited by 9
  3. 2026Evo-Bench: Can Language Models Improve Agent Harness?cited by 8
  4. 2026Hierarchical Self-Improvement: A Framework for Task-Specific Evolvable Agent Harnessescited by 6
  5. 2026SafeEvolve: Harness-Policy Co-Evolution from Agent Experience for Safety Alignmentcited by 6
  6. 2026The Last AI Built by Humans: Toward Genuine Recursive Self-Improvementcited by 5
  7. 2026RRSI: Regularized Recursive Self-Improvement of Agent Harnessescited by 5
  8. 2026Harness Continual Learning: Continual Adaptation Beyond Model Parameterscited by 4
  9. 2026Auditing Harness Tampering in Self-Improving Agentscited by 4
  10. 2026AgentStream: How Well Do Self-Evolving LLM Agents Perform Under Streaming Tasks?cited by 3
  11. 2026Graph Engineering in the Era of LLM Agents: From Individual Intelligence to System Intelligencecited by 3
  12. 2026MDArena: Evaluating Coding Agents on Realistic Molecular Dynamics Workflowscited by 2
17 more
  1. 2026Self-Evolving Coding Agentscited by 2
  2. 2026EvoSafeHarness: Evolving Model- and Domain-Specific Harnesses for Securing Agentscited by 2
  3. 2026Think-Before-Speak: From Internal Evaluation to Public Expression in Multi-Agent Social Simulationcited by 1
  4. 2026Belief-Calibrated Optimization: An Explicit World Model for Agentic Optimizationcited by 1
  5. 2026From Prompt to Harness: Coderlet from Scratchcited by 0
  6. 2026SCOPE: Score-Isolated Agentic Optimization for Video World Modelscited by 0
  7. 2026CTIFoundry: An Agent-Native Corpus Scaffold for Cyber Threat Intelligencecited by 0
  8. 2026Trojaning the Alignment: Stealthy Backdoor Attacks against Graph Foundation Modelscited by 0
  9. 2026CineForge: Self-Improving Agents for Long-Horizon Video Generationcited by 0
  10. 2026LiteEvo: Automated, Cost-Efficient Harness Evolution for Generalization to Unseen Taskscited by 0
  11. 2026EvoIn: Bridging Evolution and Internalization for Agent Fine-Tuningcited by 0
  12. 2026Harness Learning Enables Generalizable Test-Time Adaptationcited by 0
  13. 2026Self-Evolving Harness on Multiple Tasks with the Agent as Its Own Optimizercited by 0
  14. 2026From Solo to Social Learning: Characterizing Recursive Social Improvement in LLMscited by 0
  15. 2026Not All Experience Belongs in the Weights: Component Routing for Self-Improving GUI Agentscited by 0
  16. 2026Learning What to Investigate Next: Meta-Reasoning for Long-Horizon Research Agentscited by 0
  17. 2026Harness Compilation: Which Decisions Should a Small Vision-Language Model Keep?cited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Rethinking the Evaluation of Harness Evolution for Agents
  2. 2026One Recipe, Many Harnesses: What Self-Evolution Encodes Across Languages and Models ↗
  3. 2026Co-Evolving Harnesses and Models: On-Policy Correction Helps Weaker Models Catch Up Where Imitation Fails
  4. 2026Self-Harness: Harnesses That Improve Themselves
  5. 2026SEAGym: An Evaluation Environment for Self-Evolving LLM Agents
  6. 2026Adaptive Auto-Harness: Sustained Self-Improvement for Agentic System Deployment on Open-Ended Task Streams
  7. 2026EvoHarnessBench: Can Your Agents Keep Pace with an Evolving Harness?
  8. 2026HarnessForge: Joint Harness and Policy Evolution for Adaptive Agent Systems