From Question Answering to Task Completion: A Survey on Agent System and Harness Design
On the map: LLM agents and tool use → General LLM agent frameworks and benchmarks
Abstract (arXiv)
LLM-based agents mark a shift from passive question answering to active task completion: they perceive environments, invoke tools, maintain state, and act over extended horizons. As agent systems have evolved from prompt engineering to workflows and context engineering, harness engineering, and agent-native training with co-evolution, a central question has become increasingly important: where does the bottleneck in agent performance reside, in the foundation model, in the execution harness, or in the coupling between them? This survey examines LLM-based agents through a model-harness lens. We first clarify the functional definition of agents and the implementation view of an LLM-based agent as a foundation model coupled with an execution harness. We then analyze the limits of model-centric scaling, trace four paradigms of agent engineering, and decompose the execution harness into six coupled runtime responsibilities: observation, context, control, action, state, and verification. Using this decomposition, we map task properties and domain pressures to harness configurations, review benchmark and evaluation practices, and synthesize model-harness evidence on how runtime design affects long-horizon task completion, efficiency, and reliability. Finally, we identify open challenges in value-aware evaluation, safety, harness generalization, and model-harness co-evolution. Rather than treating agents as models with auxiliary tools, this survey argues that agent quality -- including success, efficiency, safety, and generalization -- emerges from the interaction between model capability, runtime infrastructure, task structure, and evaluation design. A collection of papers discussed in this survey is provided in https://github.com/ggjy/Awesome-Agent-Engineering.
Key facts. Cited by 9 papers in the map (data through 7 Oct 2026). Placed in General LLM agent frameworks and benchmarks, one of 16 directions in LLM agents and tool use. Builds on 213 of its 255 references that are in the map. First posted on arXiv 14 Jun 2026.
Builds on
213 of 255 references.
Closest ancestors · 33
Same direction, newest first
- 2026Training LLM Agents for Spontaneous, Reward-Free Self-Evolution via World Knowledge Exploration20 Apr
- 2026Agent-Driven Corpus Linguistics: A Framework for Autonomous Linguistic Discovery8 Apr
- 2026Utility-Guided Agent Orchestration for Efficient LLM Tool Use20 Mar
- 2026Brain-Inspired Graph Multi-Agent Systems for LLM Reasoning16 Mar
- 2026SAGE: Multi-Agent Self-Evolution for LLM Reasoning16 Mar
- 2026EnterpriseOps-Gym: Environments and Evaluations for Stateful Agentic Planning and Tool Use in Enterprise Settings13 Mar
- 2026From Static Benchmarks to Dynamic Protocol: Agent-Centric Text Anomaly Detection for Evaluating LLM Reasoning27 Feb
- 2026IntentCUA: Learning Intent-level Representations for Skill Abstraction and Multi-Agent Planning in Computer-Use Agents19 Feb
- 2026Differentiable Modal Logic for Multi-Agent Diagnosis, Orchestration and Communication12 Feb
- 2026From Prompt-Response to Goal-Directed Systems: The Evolution of Agentic AI Software Architecture11 Feb
- 2026Agent Alpha: Tree Search Unifying Generation, Exploration and Evaluation for Computer-Use Agents3 Feb
- 2026SYMPHONY: Synergistic Multi-agent Planning with Heterogeneous Language Model Assembly30 Jan
21 more
- 2026Meta Context Engineering via Agentic Skill Evolution29 Jan
- 2025Beyond Accuracy: A Multi-Dimensional Framework for Evaluating Enterprise Agentic AI Systems18 Nov
- 2025AgentEvolver: Towards Efficient Self-Evolving Agent System13 Nov
- 2025CostBench: Evaluating Multi-Turn Cost-Optimal Planning and Adaptation in Dynamic Environments for LLM Tool-Use Agents4 Nov
- 2025EvolveR: Self-Evolving LLM Agents through an Experience-Driven Lifecycle17 Oct
- 2025Holistic Agent Leaderboard: The Missing Infrastructure for AI Agent Evaluation13 Oct
- 2025Darwin Godel Machine: Open-Ended Evolution of Self-Improving Agents29 May
- 2025From Mind to Machine: The Rise of Manus AI as a Fully Autonomous Digital Agent4 May
- 2025Large Language Model Agent: A Survey on Methodology, Applications and Challenges27 Mar
- 2025Survey on Evaluation of LLM-based Agents20 Mar
- 2025Multi-Agent Collaboration Mechanisms: A Survey of LLMs10 Jan
- 2024TheAgentCompany: Benchmarking LLM Agents on Consequential Real World Tasks18 Dec
- 2024Magentic-One: A Generalist Multi-Agent System for Solving Complex Tasks7 Nov
- 2024WebPilot: A Versatile and Autonomous Multi-Agent System for Web Task Execution with Strategic Exploration28 Aug
- 2024MMAU: A Holistic Benchmark of Agent Capabilities Across Diverse Domains18 Jul
- 2024Tree Search for Language Model Agents1 Jul
- 2024Large Language Model based Multi-Agents: A Survey of Progress and Challenges21 Jan
- 2023The Rise and Potential of Large Language Model Based Agents: A Survey14 Sep
- 2023A Survey on Large Language Model based Autonomous Agents22 Aug
- 2023AgentBench: Evaluating LLMs as Agents7 Aug
- 2023MetaGPT: Meta Programming for A Multi-Agent Collaborative Framework1 Aug
Other references · 180
- 2020Language Models are Few-Shot Learnerscited by 32,199
- 2022Training language models to follow instructions with human feedbackcited by 15,712
- 2024The Llama 3 Herd of Modelscited by 14,192
- 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
- 2021Training Verifiers to Solve Math Word Problemscited by 9,743
- 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
- 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
- 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
- 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,355
- 2025Qwen3 Technical Reportcited by 8,195
- 2021Evaluating Large Language Models Trained on Codecited by 8,053
- 2020Measuring Massive Multitask Language Understandingcited by 7,768
38 more
- 2020Scaling Laws for Neural Language Modelscited by 6,411
- 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,601
- 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
- 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
- 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
- 2022Large Language Models are Zero-Shot Reasonerscited by 4,323
- 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,072
- 2023Reflexion: Language Agents with Verbal Reinforcement Learningcited by 4,064
- 2023Generative Agents: Interactive Simulacra of Human Behaviorcited by 3,500
- 2023Self-Refine: Iterative Refinement with Self-Feedbackcited by 3,317
- 2023SWE-bench: Can Language Models Resolve Real-World GitHub Issues?cited by 3,193
- 2023Tree of Thoughts: Deliberate Problem Solving with Large Language Modelscited by 3,155
- 2022Training Compute-Optimal Large Language Modelscited by 2,927
- 2023GPQA: A Graduate-Level Google-Proof Q&A Benchmarkcited by 2,748
- 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scalecited by 2,678
- 2023G-Eval: NLG Evaluation using GPT-4 with Better Human Alignmentcited by 2,205
- 2023Retrieval-Augmented Generation for Large Language Models: A Surveycited by 2,185
- 2023Voyager: An Open-Ended Embodied Agent with Large Language Modelscited by 2,081
- 2023WebArena: A Realistic Web Environment for Building Autonomous Agentscited by 1,908
- 2023AutoGen: Enabling Next-Gen LLM Applications via Multi-Agent Conversationcited by 1,906
- 2024MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmarkcited by 1,758
- 2024SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineeringcited by 1,628
- 2022Competition-Level Code Generation with AlphaCodecited by 1,427
- 2024From Local to Global: A Graph RAG Approach to Query-Focused Summarizationcited by 1,411
- 2024A Survey on LLM-as-a-Judgecited by 1,384
- 2022Solving Quantitative Reasoning Problems with Language Modelscited by 1,359
- 2023Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generationcited by 1,332
- 2023Gorilla: Large Language Model Connected with Massive APIscited by 1,302
- 2023MemGPT: Towards LLMs as Operating Systemscited by 1,295
- 2021Improving language models by retrieving from trillions of tokenscited by 1,278
- 2020Leveraging Passage Retrieval with Generative Models for Open Domain Question Answeringcited by 1,203
- 2024OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environmentscited by 1,134
- 2023GAIA: a benchmark for General AI Assistantscited by 1,084
- 2024OpenHands: An Open Platform for AI Software Developers as Generalist Agentscited by 956
- 2022Atlas: Few-shot Learning with Retrieval Augmented Language Modelscited by 940
- 2022CodeGen: An Open Large Language Model for Code with Multi-Turn Program Synthesiscited by 876
- 2024Evaluating Very Long-Term Conversational Memory of LLM Agentscited by 800
- 2024BigCodeBench: Benchmarking Code Generation with Diverse Function Calls and Complex Instructionscited by 527
50 most cited of 180 All 180 (JSON)
Built on by
Citing papers per year: 2026: 9.
9 papers.
- 2026Agentic Routing: The Harness-Native Data Flywheelcited by 4
- 2026CHILL-Harness: Counterfactual Harness Learning for Efficient Reasoning in Long-Horizon Agentscited by 1
- 2026OneDayAgent: Towards a Long-Horizon Harness for Autonomous Agentscited by 1
- 2026A Responsive Present, a Shared Past, a Social Other: Teens' Overreliance on Companion AI Chatbotscited by 0
- 2026GraphSkillEvo: Evolutionary Optimization of Graph-Structured Agent Skillscited by 0
- 2026Distilling Agentic Systems: A Roadmap across Models, Artifacts, and Harnessescited by 0
- 2026AgentBug-Smith: Automatically Reproducing Real-World Harness Bugs in Agentic Systemscited by 0
- 2026VidHarness: Evolving Agent Harnesses for Cost-Efficient Long Video Understandingcited by 0
- 2026JusticeAxis: Benchmarking Legal Judgment between Rigid Rule Application and Ungrounded Discretioncited by 0
- 2026JusticeAxis: Benchmarking Legal Judgment between Rigid Rule Application and Ungrounded Discretion30 Sep 2026
- 2026VidHarness: Evolving Agent Harnesses for Cost-Efficient Long Video Understanding29 Sep 2026
- 2026AgentBug-Smith: Automatically Reproducing Real-World Harness Bugs in Agentic Systems29 Sep 2026
- 2026Distilling Agentic Systems: A Roadmap across Models, Artifacts, and Harnesses29 Sep 2026
- 2026GraphSkillEvo: Evolutionary Optimization of Graph-Structured Agent Skills18 Sep 2026
- 2026A Responsive Present, a Shared Past, a Social Other: Teens' Overreliance on Companion AI Chatbots13 Sep 2026
- 2026OneDayAgent: Towards a Long-Horizon Harness for Autonomous Agents4 Aug 2026
- 2026CHILL-Harness: Counterfactual Harness Learning for Efficient Reasoning in Long-Horizon Agents28 Jul 2026
- 2026Agentic Routing: The Harness-Native Data Flywheel13 Jul 2026
Similar, not cited
Close in wording, no citation either way
- 2024AgentQuest: A Modular Benchmark Framework to Measure Progress and Improve LLM Agents
- 2026LongHorizon-Harness: Advancing Long-Horizon Agents for Real-World Tasks
- 2026AgentGym2: Benchmarking Large Language Model Agents in De-Idealized Real-World Environments
- 2026A Comprehensive Survey on Agent Skills: Taxonomy, Techniques, and Applications
- 2025Agent-Based Detection and Resolution of Incompleteness and Ambiguity in Interactions with Large Language Models
- 2025Beyond Reactivity: Measuring Proactive Problem Solving in LLM Agents
- 2026Externalization in LLM Agents: A Unified Review of Memory, Skills, Protocols and Harness Engineering
- 2026SKILL.state: Scalable Long-Horizon Agent Skills