mlsanityData to 7 Oct 2026

Code as Agent Harness

arXiv 2605.18747 ↗First posted 18 May 2026cs.CL cs.AIcited by 40

On the map: LLM agents and tool use → Software engineering coding agents

Abstract (arXiv)

Recent large language models (LLMs) have demonstrated strong capabilities in understanding and generating code, from competitive programming to repository-level software engineering. In emerging agentic systems, code is no longer only a target output. It increasingly serves as an operational substrate for agent reasoning, acting, environment modeling, and execution-based verification. We frame this shift through the lens of agent harnesses and introduce code as agent harness: a unified view that centers code as the basis for agent infrastructure. To systematically study this perspective, we organize the survey around three connected layers. First, we study the harness interface, where code connects agents to reasoning, action, and environment modeling. Second, we examine harness mechanisms: planning, memory, and tool use for long-horizon execution, together with feedback-driven control and optimization that make harness reliable and adaptive. Third, we discuss scaling the harness from single-agent systems to multi-agent settings, where shared code artifacts support multi-agent coordination, review, and verification. Across these layers, we summarize representative methods and practical applications of code as agent harness, spanning coding assistants, GUI/OS automation, embodied agents, scientific discovery, personalization and recommendation, DevOps, and enterprise workflows. We further outline open challenges for harness engineering, including evaluation beyond final task success, verification under incomplete feedback, regression-free harness improvement, consistent shared state across multiple agents, human oversight for safety-critical actions, and extensions to multimodal environments. By centering code as the harness of agentic AI, this survey provides a unified roadmap toward executable, verifiable, and stateful AI agent systems.

Key facts. Cited by 40 papers in the map (data through 7 Oct 2026). Placed in Software engineering coding agents, one of 16 directions in LLM agents and tool use. Builds on 234 of its 478 references that are in the map. First posted on arXiv 18 May 2026.

Builds on

234 of 478 references.

Closest ancestors · 46

Same direction, newest first

  1. 2026Agentic Harness Engineering: Observability-Driven Automatic Evolution of Coding-Agent Harnesses28 Apr
  2. 2026Dive into Claude Code: The Design Space of Today's and Future AI Agent Systems14 Apr
  3. 2026Learning to Commit: Generating Organic Pull Requests via Online Repository Memory27 Mar
  4. 2026CodeTaste: Can LLMs Generate Human-Level Code Refactorings?4 Mar
  5. 2026SGAgent: Suggestion-Guided LLM-Based Multi-Agent Framework for Repository-Level Software Repair27 Feb
  6. 2026Your Code Agent Can Grow Alongside You with Structured Memory25 Feb
  7. 2026LongCLI-Bench: A Preliminary Benchmark and Study for Long-horizon Agentic Programming in Command-Line Interfaces15 Feb
  8. 2026Environment-in-the-Loop: Rethinking Code Migration with LLM-based Agents10 Feb
  9. 2026ContextBench: A Benchmark for Context Retrieval in Coding Agents5 Feb
  10. 2026SWE-Pruner: Self-Adaptive Context Pruning for Coding Agents23 Jan
  11. 2025BOAD: Discovering Hierarchical Software Engineering Agents via Bandit Optimization29 Dec
  12. 2025Live-SWE-agent: Can Software Engineering Agents Self-Evolve on the Fly?17 Nov
34 more
  1. 2025The OpenHands Software Agent SDK: A Composable and Extensible Foundation for Production Agents5 Nov
  2. 2025TALM: Dynamic Tree-Structured Multi-Agent Framework with Long-Term Memory for Scalable Code Generation27 Oct
  3. 2025Multi-Agent Code-Orchestrated Generation for Reliable Infrastructure-as-Code4 Oct
  4. 2025Where LLM Agents Fail and How They can Learn From Failures29 Sep
  5. 2025AgenTracer: Who Is Inducing Failure in the LLM Agentic Systems?3 Sep
  6. 2025Requirements Development and Formalization for Reliable Code Generation: A Multi-Agent Vision26 Aug
  7. 2025Self-Organizing Agent Network for LLM-based Workflow Automation19 Aug
  8. 2025CP-Agent: Agentic Constraint Programming10 Aug
  9. 2025A Survey on Code Generation with LLM-based Agents31 Jul
  10. 2025Trae Agent: An LLM-based Agent for Software Engineering with Test-time Scaling31 Jul
  11. 2025SWE-Debate: Competitive Multi-Agent Debate for Software Issue Resolution31 Jul
  12. 2025AlphaEvolve: A coding agent for scientific and algorithmic discovery16 Jun
  13. 2025ReVeal: Self-Evolving Code Agents via Reliable Self-Verification13 Jun
  14. 2025Hallucination to Consensus: Multi-Agent LLMs for End-to-End JUnit Test Generation3 Jun
  15. 2025Vibe Coding vs. Agentic Coding: Fundamentals and Practical Implications of Agentic AI26 May
  16. 2025SEW: Self-Evolving Agentic Workflows for Automated Code Generation24 May
  17. 2025MARCO: Multi-Agent Code Optimization with Real-Time Knowledge Integration for High-Performance Computing6 May
  18. 2025Which Agent Causes Task Failures and When? On Automated Failure Attribution of LLM Multi-Agent Systems30 Apr
  19. 2025SWE-smith: Scaling Data for Software Engineering Agents30 Apr
  20. 2025GraphCodeAgent: Dual Graph-Guided LLM Agent for Retrieval-Augmented Repo-Level Code Generation14 Apr
  21. 2025Why Do Multi-Agent LLM Systems Fail?17 Mar
  22. 2025QualityFlow: An Agentic Workflow for Program Synthesis Controlled by LLM Quality Checks20 Jan
  23. 2025CodeCoR: An LLM-Based Self-Reflective Multi-Agent Framework for Code Generation14 Jan
  24. 2024MAGE: A Multi-Agent Engine for Automated RTL Code Generation10 Dec
  25. 2024Lingma SWE-GPT: An Open Development-Process-Centric Language Model for Automated Software Improvement1 Nov
  26. 2024SWE-Search: Enhancing Software Agents with Monte Carlo Tree Search and Iterative Refinement26 Oct
  27. 2024HyperAgent: Generalist Software Engineering Agents to Solve Coding Tasks at Scale9 Sep
  28. 2024Towards Agentic Runtime Healing2 Aug
  29. 2024Agentless: Demystifying LLM-based Software Engineering Agents1 Jul
  30. 2024Self-Organized Agents: A LLM Multi-Agent Framework toward Ultra Large-Scale Code Generation and Optimization2 Apr
  31. 2024CodePori: Large-Scale System for Autonomous Software Development Using Multi-Agent Technology2 Feb
  32. 2024Executable Code Actions Elicit Better LLM Agents1 Feb
  33. 2023AgentCoder: Multi-Agent-based Code Generation with Iterative Testing and Optimisation20 Dec
  34. 2023GameGPT: Multi-agent Collaborative Framework for Game Development12 Oct

Other references · 188

  1. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,355
  2. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  3. 2021Program Synthesis with Large Language Modelscited by 3,509
  4. 2023SWE-bench: Can Language Models Resolve Real-World GitHub Issues?cited by 3,193
  5. 2022Do As I Can, Not As I Say: Grounding Language in Robotic Affordancescited by 2,302
  6. 2023PaLM-E: An Embodied Multimodal Language Modelcited by 2,037
  7. 2023WebArena: A Realistic Web Environment for Building Autonomous Agentscited by 1,908
  8. 2024LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Codecited by 1,906
  9. 2023MemGPT: Towards LLMs as Operating Systemscited by 1,295
  10. 2023AgentBench: Evaluating LLMs as Agentscited by 1,211
  11. 2023Mind2Web: Towards a Generalist Agent for the Webcited by 1,210
  12. 2022WebShop: Towards Scalable Real-World Web Interaction with Grounded Language Agentscited by 1,197
38 more
  1. 2022Code as Policies: Language Model Programs for Embodied Controlcited by 1,172
  2. 2024OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environmentscited by 1,134
  3. 2022Inner Monologue: Embodied Reasoning through Planning with Language Modelscited by 1,057
  4. 2022Program of Thoughts Prompting: Disentangling Computation from Reasoning for Numerical Reasoning Taskscited by 1,057
  5. 2024The AI Scientist: Towards Fully Automated Open-Ended Scientific Discoverycited by 1,002
  6. 2023Teaching Large Language Models to Self-Debugcited by 963
  7. 2022CodeGen: An Open Large Language Model for Code with Multi-Turn Program Synthesiscited by 876
  8. 2023VoxPoser: Composable 3D Value Maps for Robotic Manipulation with Language Modelscited by 733
  9. 2025Mem0: Building Production-Ready AI Agents with Scalable Long-Term Memorycited by 722
  10. 2023CogAgent: A Visual Language Model for GUI Agentscited by 692
  11. 2017DeepFM: A Factorization-Machine based Neural Network for CTR Predictioncited by 673
  12. 2023ChemCrow: Augmenting large-language models with chemistry toolscited by 609
  13. 2025UI-TARS: Pioneering Automated GUI Interaction with Native Agentscited by 584
  14. 2024SeeClick: Harnessing GUI Grounding for Advanced Visual GUI Agentscited by 553
  15. 2024GPT-4V(ision) is a Generalist Web Agent, if Groundedcited by 547
  16. 2026Terminal-Bench: Benchmarking Agents on Hard, Realistic Tasks in Command Line Interfacescited by 515
  17. 2024AndroidWorld: A Dynamic Benchmarking Environment for Autonomous Agentscited by 450
  18. 2025GEPA: Reflective Prompt Evolution Can Outperform Reinforcement Learningcited by 449
  19. 2024MLE-bench: Evaluating Machine Learning Agents on Machine Learning Engineeringcited by 429
  20. 2024WebVoyager: Building an End-to-End Web Agent with Large Multimodal Modelscited by 424
  21. 2023AppAgent: Multimodal Agents as Smartphone Userscited by 407
  22. 2025Agent Laboratory: Using LLM Agents as Research Assistantscited by 405
  23. 2025The AI Scientist-v2: Workshop-Level Automated Scientific Discovery via Agentic Tree Searchcited by 375
  24. 2024Navigating the Digital World as Humans Do: Universal Visual Grounding for GUI Agentscited by 373
  25. 2024OS-ATLAS: A Foundation Action Model for Generalist GUI Agentscited by 370
  26. 2025Accelerating scientific discovery with Co-Scientistcited by 353
  27. 2023A Real-World WebAgent with Planning, Long Context Understanding, and Program Synthesiscited by 350
  28. 2025Agentic Context Engineering: Evolving Contexts for Self-Improving Language Modelscited by 349
  29. 2024WorkArena: How Capable Are Web Agents at Solving Common Knowledge Work Tasks?cited by 348
  30. 2023MLAgentBench: Evaluating Language Agents on Machine Learning Experimentationcited by 339
  31. 2024TheAgentCompany: Benchmarking LLM Agents on Consequential Real World Taskscited by 306
  32. 2023Robots That Ask For Help: Uncertainty Alignment for Large Language Model Plannerscited by 298
  33. 2024CRUXEval: A Benchmark for Code Reasoning, Understanding and Executioncited by 294
  34. 2023Set-of-Mark Prompting Unleashes Extraordinary Visual Grounding in GPT-4Vcited by 291
  35. 2018Reinforcement Learning on Web Interfaces Using Workflow-Guided Explorationcited by 282
  36. 2025DeepSeek-Prover-V2: Advancing Formal Mathematical Reasoning via Reinforcement Learning for Subgoal Decompositioncited by 270
  37. 2023The Impact of AI on Developer Productivity: Evidence from GitHub Copilotcited by 264
  38. 2025SWE-Bench Pro: Can AI Agents Solve Long-Horizon Software Engineering Tasks?cited by 254

50 most cited of 188 All 188 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 40.

40 papers.

Sort:

By area:

  1. 2026Harness VLA: Steering Frozen VLAs into Reliable Manipulation Primitives via Memory-Guided Agentscited by 37
  2. 2026Harness Updating Is Not Harness Benefit: Disentangling Evolution Capabilities in Self-Evolving LLM Agentscited by 28
  3. 2026SkillOpt-Lite: Better and Faster Agent Self-evolution via One Line of Vibecited by 18
  4. 2026Harness Handbook: Making Evolving Agent Harnesses Readable,Navigable, and Editablecited by 12
  5. 2026JIT-Agent: Scaling Harness Intelligence via Just-in-Time Harness Evolutioncited by 10
  6. 2026From Question Answering to Task Completion: A Survey on Agent System and Harness Designcited by 9
  7. 2026From Model Scaling to System Scaling: Scaling the Harness in Agentic AIcited by 8
  8. 2026HarnessDev: Can LLMs Create and Evolve Their Own Agent Harness?cited by 7
  9. 2026AI4AI at Test-Time: Strong-to-Weak Capability Transfer via Harnessescited by 5
  10. 2026Automated Discovery Has No Universally Superior Harnesscited by 4
  11. 2026DREvo: Distilling Recalibrated Historical Experience for Harness Self-Evolutioncited by 4
  12. 2026From Prompts to Contracts: Harness Engineering for Auditable Enterprise LLM Agentscited by 3
28 more
  1. 2026Harness-Zero: Harness Distillation via Agent-as-Harnesscited by 3
  2. 2026ReContext: Recursive Evidence Replay as LLM Harness for Long-Context Reasoningcited by 2
  3. 2026Beyond the Best Guess: Improving LLM Solution Coverage with Evolution Strategiescited by 2
  4. 2026SoL-Pi: Recursively Scaling Auto-Research Loops for Efficient Agent Harnesscited by 2
  5. 2026SearchOS-V1: Towards Robust Open-Domain Information-Seeking Agent Collaborationcited by 1
  6. 2026EvoHarness-RL: Learning Self-Evolving Runtime Harness for Long-Horizon LLM Agentscited by 1
  7. 2026Training Agents to Evolve with Their Harness: TaoLive Digital Avatar Agent Technical Reportcited by 1
  8. 2026Graph Engineering in the Era of LLM Agents: From Individual Intelligence to System Intelligencecited by 1
  9. 2026Harness Engineering: Anatomy, Architecture, and Evolution of Coding Agents -- A Source-Code Study of Eleven Systemscited by 1
  10. 2026From Language Models to World-Acting Systems: Progress and Limits of Agentic AI across Digital, Social, Virtual, and Physical Environmentscited by 1
  11. 2026Inference-Time Graph Engineering for Multi-Agent LLM Workflowscited by 1
  12. 2026MUSE: A Unified Agentic Harness for MLLMscited by 0
  13. 2026Claim Plane: Reliability Gains and the Limits of Selective Concurrency for Parallel Coding Agents: A 30-Pair, Three-Seed Confirmatory Study of Deterministic Pre-Write Admissioncited by 0
  14. 2026ADIAS: Automated Design of Interactive Agentic Systemscited by 0
  15. 2026From Prompt to Harness: Coderlet from Scratchcited by 0
  16. 2026Does a Language Server Save Tokens for Coding Agents? A Measurement Methodology and Preliminary Studycited by 0
  17. 2026Resource Constraints and Performance in Agentic AI Systemscited by 0
  18. 2026DS-Lighting: Making Agent Harnesses Explicit for Data-Science Automationcited by 0
  19. 2026WeAgent-MMGenEdit: A Full-Stack Recipe for Multimodal Agentic Image Generation and Editingcited by 0
  20. 2026PluginRSI: Recursive Improvement of Agent Harnesses with Reusable Pluginscited by 0
  21. 2026MaLiang-Harness: A Programmable Path to Image and Video Generationcited by 0
  22. 2026EvoIn: Bridging Evolution and Internalization for Agent Fine-Tuningcited by 0
  23. 2026Act First, Reason Later: Accelerating On-Policy Distillation for Multi-Turn Agents via Reference-Conditioned Inverse Dynamicscited by 0
  24. 2026Video2Skill: From Streaming Experience to Reusable Embodied Skillscited by 0
  25. 2026Neuro-Symbolic Computer Use: Learning Reusable Policies for Reliable and Efficient Executioncited by 0
  26. 2026AgentBug-Smith: Automatically Reproducing Real-World Harness Bugs in Agentic Systemscited by 0
  27. 2026Learning from Research: Toward Lifelong Agent Harness Evolutioncited by 0
  28. 2026Safe, Persistent, and Evolving Agent Harness for Understanding Partially Observable Worldscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026An Empirical Study of Harness Design for Coding Agents
  2. 2026Don't Blame the Large Language Model: How Agent Harness Evolution Shapes Coding Agent Quality
  3. 2026Composing Task-specific Agent Harnesses at Test Time with Reusable Primitives
  4. 2026The Empire, Long Divided, Must Unite: Architectural Convergence in Three LLM Agent Harnesses
  5. 2026Inside the Scaffold: A Source-Code Taxonomy of Coding Agent Architectures
  6. 2026openJiuwen: Beyond Static Harnesses for Long-Horizon Coding Agents
  7. 2026Harness Engineering for Agentic AI Coding Tools: An Exploratory Study
  8. 2026Harness-Bench: Measuring Harness Effects across Models in Realistic Agent Workflows