mlsanityData to 14 Oct 2026

Resource Constraints and Performance in Agentic AI Systems

arXiv 2608.27886 ↗First posted 28 Aug 2026cs.AIcited by 0

On the map: LLM agents and tool use → General LLM agent frameworks and benchmarks

Abstract (arXiv)

Progress toward more autonomous AI increasingly depends on agentic systems that combine a language model with tools, memory, state management, and multi-step execution. These mechanisms shape both task capability and operational burden. We compare OpenClaw and NanoBot as complete agentic systems using a paired primary benchmark and a more detailed instrumented subset of paired prompts. In the primary benchmark, the rate of full task completion was 31% for OpenClaw and 25% for NanoBot, a six-percentage-point difference with a 95% task-bootstrap interval from -3 to 15 percentage points, providing no statistically established full-completion advantage for either system. In the instrumented layer, both systems achieved 26% full completion, while NanoBot reached at least partial completion on 43% of prompts compared with 26% for OpenClaw. OpenClaw took longer on 83% of prompts and had a higher recorded peak-memory value on every prompt, with geometric mean ratios of 2.98 for wall time and 19.44 for peak memory. Among the ten detailed-layer prompts on which at least one system achieved partial or full completion, NanoBot weakly dominated on eight; across all 23 prompts, however, ten of its eighteen dominance cases were cheaper joint failures. Outcome labels differ across the two evidence layers, showing why agent-system evaluation should connect capability and resource measurements to attempt-level execution and scoring provenance. These findings show that progress toward more autonomous AI should be evaluated through verified task completion, observed resource use and records linking each result to the execution that produced it.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in General LLM agent frameworks and benchmarks, one of 16 directions in LLM agents and tool use. Builds on 31 of its 52 references that are in the map. First posted on arXiv 28 Aug 2026.

Builds on

31 of 52 references.

Closest ancestors · 11

Same direction, newest first

  1. 2026Evaluating Agentic AI in the Wild: Failure Modes, Drift Patterns, and a Production Evaluation Framework2 May
  2. 2026Architectural Design Decisions in AI Agent Harnesses20 Apr
  3. 2026MASEval: Extending Multi-Agent Evaluation from Models to Systems9 Mar
  4. 2026Architectural Design, Not Only Model Intelligence, Governs Multi-Agent LLM Performance3 Feb
  5. 2025Generalizability of Large Language Model-Based Agents: A Comprehensive Survey19 Sep
  6. 2025Evaluation and Benchmarking of LLM Agents: A Survey29 Jul
  7. 2025The Cost of Dynamic Reasoning: Demystifying AI Agents and Test-Time Scaling from an AI Infrastructure Perspective4 Jun
  8. 2025Survey on Evaluation of LLM-based Agents20 Mar
  9. 2025A Survey on the Optimization of Large Language Model-based Agents16 Mar
  10. 2024τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains17 Jun
  11. 2023AgentBench: Evaluating LLMs as Agents7 Aug

Other references · 20

  1. 2023WebArena: A Realistic Web Environment for Building Autonomous Agentscited by 2,006
  2. 2024SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineeringcited by 1,737
  3. 2024OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environmentscited by 1,195
  4. 2023GAIA: a benchmark for General AI Assistantscited by 1,128
  5. 2024OpenHands: An Open Platform for AI Software Developers as Generalist Agentscited by 1,002
  6. 2023Reasoning with Language Model is Planning with World Modelcited by 861
  7. 2025Why Do Multi-Agent LLM Systems Fail?cited by 604
  8. 2022PlanBench: An Extensible Benchmark for Evaluating Large Language Models on Planning and Reasoning about Changecited by 382
  9. 2023On the Planning Abilities of Large Language Models : A Critical Investigationcited by 312
  10. 2024AI Agents That Mattercited by 219
  11. 2025Measuring AI Ability to Complete Long Software Taskscited by 156
  12. 2024The Emerged Security and Privacy of LLM Agent: A Survey with Case Studiescited by 131
8 more
  1. 2023CLadder: Assessing Causal Reasoning in Language Modelscited by 107
  2. 2025Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivitycited by 101
  3. 2023Levels of AGI for Operationalizing Progress on the Path to AGIcited by 96
  4. 2025A Survey on Trustworthy LLM Agents: Threats and Countermeasurescited by 96
  5. 2026Towards a Science of AI Agent Reliabilitycited by 65
  6. 2026Code as Agent Harnesscited by 47
  7. 2025REAL: Benchmarking Autonomous Agents on Deterministic Simulations of Real Websitescited by 45
  8. 2026Think Fast: Estimating No-CoT Task-Completion Time Horizons of Frontier AI Modelscited by 12

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026OpenClaw and Ollama in Agentic AI: Toward Fully Autonomous and Scalable AI Agent Systems
  2. 2025Exploring Autonomous Agents: A Closer Look at Why They Fail When Completing Tasks
  3. 2025Beyond Task Completion: An Assessment Framework for Evaluating Agentic AI Systems
  4. 2025AgentArch: A Comprehensive Benchmark to Evaluate Agent Architectures in Enterprise
  5. 2025Beyond Black-Box Benchmarking: Observability, Analytics, and Optimization of Agentic Systems
  6. 2026The Path Ahead for Agentic AI: Challenges and Opportunities
  7. 2025Towards a Science of Scaling Agent Systems
  8. 2026AgencyBench: Benchmarking the Frontiers of Autonomous Agents in 1M-Token Real-World Contexts