mlsanityData to 7 Oct 2026

From Question Answering to Task Completion: A Survey on Agent System and Harness Design

arXiv 2606.20683 ↗First posted 14 Jun 2026cs.AI cs.CLcited by 9

On the map: LLM agents and tool use → General LLM agent frameworks and benchmarks

Abstract (arXiv)

LLM-based agents mark a shift from passive question answering to active task completion: they perceive environments, invoke tools, maintain state, and act over extended horizons. As agent systems have evolved from prompt engineering to workflows and context engineering, harness engineering, and agent-native training with co-evolution, a central question has become increasingly important: where does the bottleneck in agent performance reside, in the foundation model, in the execution harness, or in the coupling between them? This survey examines LLM-based agents through a model-harness lens. We first clarify the functional definition of agents and the implementation view of an LLM-based agent as a foundation model coupled with an execution harness. We then analyze the limits of model-centric scaling, trace four paradigms of agent engineering, and decompose the execution harness into six coupled runtime responsibilities: observation, context, control, action, state, and verification. Using this decomposition, we map task properties and domain pressures to harness configurations, review benchmark and evaluation practices, and synthesize model-harness evidence on how runtime design affects long-horizon task completion, efficiency, and reliability. Finally, we identify open challenges in value-aware evaluation, safety, harness generalization, and model-harness co-evolution. Rather than treating agents as models with auxiliary tools, this survey argues that agent quality -- including success, efficiency, safety, and generalization -- emerges from the interaction between model capability, runtime infrastructure, task structure, and evaluation design. A collection of papers discussed in this survey is provided in https://github.com/ggjy/Awesome-Agent-Engineering.

Key facts. Cited by 9 papers in the map (data through 7 Oct 2026). Placed in General LLM agent frameworks and benchmarks, one of 16 directions in LLM agents and tool use. Builds on 213 of its 255 references that are in the map. First posted on arXiv 14 Jun 2026.

Builds on

213 of 255 references.

Closest ancestors · 33

Same direction, newest first

  1. 2026Training LLM Agents for Spontaneous, Reward-Free Self-Evolution via World Knowledge Exploration20 Apr
  2. 2026Agent-Driven Corpus Linguistics: A Framework for Autonomous Linguistic Discovery8 Apr
  3. 2026Utility-Guided Agent Orchestration for Efficient LLM Tool Use20 Mar
  4. 2026Brain-Inspired Graph Multi-Agent Systems for LLM Reasoning16 Mar
  5. 2026SAGE: Multi-Agent Self-Evolution for LLM Reasoning16 Mar
  6. 2026EnterpriseOps-Gym: Environments and Evaluations for Stateful Agentic Planning and Tool Use in Enterprise Settings13 Mar
  7. 2026From Static Benchmarks to Dynamic Protocol: Agent-Centric Text Anomaly Detection for Evaluating LLM Reasoning27 Feb
  8. 2026IntentCUA: Learning Intent-level Representations for Skill Abstraction and Multi-Agent Planning in Computer-Use Agents19 Feb
  9. 2026Differentiable Modal Logic for Multi-Agent Diagnosis, Orchestration and Communication12 Feb
  10. 2026From Prompt-Response to Goal-Directed Systems: The Evolution of Agentic AI Software Architecture11 Feb
  11. 2026Agent Alpha: Tree Search Unifying Generation, Exploration and Evaluation for Computer-Use Agents3 Feb
  12. 2026SYMPHONY: Synergistic Multi-agent Planning with Heterogeneous Language Model Assembly30 Jan
21 more
  1. 2026Meta Context Engineering via Agentic Skill Evolution29 Jan
  2. 2025Beyond Accuracy: A Multi-Dimensional Framework for Evaluating Enterprise Agentic AI Systems18 Nov
  3. 2025AgentEvolver: Towards Efficient Self-Evolving Agent System13 Nov
  4. 2025CostBench: Evaluating Multi-Turn Cost-Optimal Planning and Adaptation in Dynamic Environments for LLM Tool-Use Agents4 Nov
  5. 2025EvolveR: Self-Evolving LLM Agents through an Experience-Driven Lifecycle17 Oct
  6. 2025Holistic Agent Leaderboard: The Missing Infrastructure for AI Agent Evaluation13 Oct
  7. 2025Darwin Godel Machine: Open-Ended Evolution of Self-Improving Agents29 May
  8. 2025From Mind to Machine: The Rise of Manus AI as a Fully Autonomous Digital Agent4 May
  9. 2025Large Language Model Agent: A Survey on Methodology, Applications and Challenges27 Mar
  10. 2025Survey on Evaluation of LLM-based Agents20 Mar
  11. 2025Multi-Agent Collaboration Mechanisms: A Survey of LLMs10 Jan
  12. 2024TheAgentCompany: Benchmarking LLM Agents on Consequential Real World Tasks18 Dec
  13. 2024Magentic-One: A Generalist Multi-Agent System for Solving Complex Tasks7 Nov
  14. 2024WebPilot: A Versatile and Autonomous Multi-Agent System for Web Task Execution with Strategic Exploration28 Aug
  15. 2024MMAU: A Holistic Benchmark of Agent Capabilities Across Diverse Domains18 Jul
  16. 2024Tree Search for Language Model Agents1 Jul
  17. 2024Large Language Model based Multi-Agents: A Survey of Progress and Challenges21 Jan
  18. 2023The Rise and Potential of Large Language Model Based Agents: A Survey14 Sep
  19. 2023A Survey on Large Language Model based Autonomous Agents22 Aug
  20. 2023AgentBench: Evaluating LLMs as Agents7 Aug
  21. 2023MetaGPT: Meta Programming for A Multi-Agent Collaborative Framework1 Aug

Other references · 180

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2022Training language models to follow instructions with human feedbackcited by 15,712
  3. 2024The Llama 3 Herd of Modelscited by 14,192
  4. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  5. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  6. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
  7. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
  8. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  9. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,355
  10. 2025Qwen3 Technical Reportcited by 8,195
  11. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  12. 2020Measuring Massive Multitask Language Understandingcited by 7,768
38 more
  1. 2020Scaling Laws for Neural Language Modelscited by 6,411
  2. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,601
  3. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  4. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
  5. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  6. 2022Large Language Models are Zero-Shot Reasonerscited by 4,323
  7. 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,072
  8. 2023Reflexion: Language Agents with Verbal Reinforcement Learningcited by 4,064
  9. 2023Generative Agents: Interactive Simulacra of Human Behaviorcited by 3,500
  10. 2023Self-Refine: Iterative Refinement with Self-Feedbackcited by 3,317
  11. 2023SWE-bench: Can Language Models Resolve Real-World GitHub Issues?cited by 3,193
  12. 2023Tree of Thoughts: Deliberate Problem Solving with Large Language Modelscited by 3,155
  13. 2022Training Compute-Optimal Large Language Modelscited by 2,927
  14. 2023GPQA: A Graduate-Level Google-Proof Q&A Benchmarkcited by 2,748
  15. 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scalecited by 2,678
  16. 2023G-Eval: NLG Evaluation using GPT-4 with Better Human Alignmentcited by 2,205
  17. 2023Retrieval-Augmented Generation for Large Language Models: A Surveycited by 2,185
  18. 2023Voyager: An Open-Ended Embodied Agent with Large Language Modelscited by 2,081
  19. 2023WebArena: A Realistic Web Environment for Building Autonomous Agentscited by 1,908
  20. 2023AutoGen: Enabling Next-Gen LLM Applications via Multi-Agent Conversationcited by 1,906
  21. 2024MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmarkcited by 1,758
  22. 2024SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineeringcited by 1,628
  23. 2022Competition-Level Code Generation with AlphaCodecited by 1,427
  24. 2024From Local to Global: A Graph RAG Approach to Query-Focused Summarizationcited by 1,411
  25. 2024A Survey on LLM-as-a-Judgecited by 1,384
  26. 2022Solving Quantitative Reasoning Problems with Language Modelscited by 1,359
  27. 2023Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generationcited by 1,332
  28. 2023Gorilla: Large Language Model Connected with Massive APIscited by 1,302
  29. 2023MemGPT: Towards LLMs as Operating Systemscited by 1,295
  30. 2021Improving language models by retrieving from trillions of tokenscited by 1,278
  31. 2020Leveraging Passage Retrieval with Generative Models for Open Domain Question Answeringcited by 1,203
  32. 2024OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environmentscited by 1,134
  33. 2023GAIA: a benchmark for General AI Assistantscited by 1,084
  34. 2024OpenHands: An Open Platform for AI Software Developers as Generalist Agentscited by 956
  35. 2022Atlas: Few-shot Learning with Retrieval Augmented Language Modelscited by 940
  36. 2022CodeGen: An Open Large Language Model for Code with Multi-Turn Program Synthesiscited by 876
  37. 2024Evaluating Very Long-Term Conversational Memory of LLM Agentscited by 800
  38. 2024BigCodeBench: Benchmarking Code Generation with Diverse Function Calls and Complex Instructionscited by 527

50 most cited of 180 All 180 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 9.

9 papers.

Similar, not cited

Close in wording, no citation either way

  1. 2024AgentQuest: A Modular Benchmark Framework to Measure Progress and Improve LLM Agents
  2. 2026LongHorizon-Harness: Advancing Long-Horizon Agents for Real-World Tasks
  3. 2026AgentGym2: Benchmarking Large Language Model Agents in De-Idealized Real-World Environments
  4. 2026A Comprehensive Survey on Agent Skills: Taxonomy, Techniques, and Applications
  5. 2025Agent-Based Detection and Resolution of Incompleteness and Ambiguity in Interactions with Large Language Models
  6. 2025Beyond Reactivity: Measuring Proactive Problem Solving in LLM Agents
  7. 2026Externalization in LLM Agents: A Unified Review of Memory, Skills, Protocols and Harness Engineering
  8. 2026SKILL.state: Scalable Long-Horizon Agent Skills