AgentSLABench: Evaluating and Benchmarking Agentic Systems Under Resource Constraints

arXiv 2608.00805 ↗First posted 1 Aug 2026cs.AIcited by 0

On the map: LLM agents and tool use → General LLM agent frameworks and benchmarks

Abstract (arXiv)

We present AgentSLABench, a resource-aware evaluation framework for autonomous AI agents that measures correctness alongside latency, cost, compute, memory, and network usage under declared resource budgets. Unlike standard benchmarks that report only accuracy, AgentSLABench produces a multi-dimensional profile per agent per task - the same way systems profilers (perf, pprof, cProfile) measure resource consumption of code, but extended with task correctness as a first-class dimension. AgentSLABench provides 16 task environments across 6 categories (5 core: multi-hop QA, retail substitution, code generation, web shopping, travel planning; 11 extended) with isolated Docker containers, declared CPU/memory/time/network budgets, sealed test sets with SHA256 hashes, and a standardized profiling protocol. We profile 5 general-purpose baseline agents (ReAct, PlanAndSolve, Reflexion, CoT, Random) plus 4 task-specialized agents, finding that specialized agents achieve 100% success on 3/5 core tasks (fact_qa, web_shopping, travel_planning) and 66.7-83.3% on retail and code_gen, while general baselines fail entirely on 4/5 domain tasks. Crucially, we report the Efficiency-Adjusted Success Rate (EASR) - success weighted by resource consumption relative to declared budgets - revealing that high accuracy at unbounded cost is not production-viable. We release the full infrastructure, sealed test sets, and profiling results to enable reproducible, resource-aware agent evaluation.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in General LLM agent frameworks and benchmarks, one of 16 directions in LLM agents and tool use. Builds on 18 of its 25 references that are in the map. First posted on arXiv 1 Aug 2026.

Builds on

18 of 25 references.

Closest ancestors · 1

Same direction, newest first

2023AgentBench: Evaluating LLMs as Agents7 Aug

Other references · 17

2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,695
2023Reflexion: Language Agents with Verbal Reinforcement Learningcited by 4,276
2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,148
2023SWE-bench: Can Language Models Resolve Real-World GitHub Issues?cited by 3,373
2023AutoGen: Enabling Next-Gen LLM Applications via Multi-Agent Conversationcited by 1,988
2023ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIscited by 1,947
2022WebShop: Towards Scalable Real-World Web Interaction with Grounded Language Agentscited by 1,251
2020ALFWorld: Aligning Text and Embodied Environments for Interactive Learningcited by 1,205
2023GAIA: a benchmark for General AI Assistantscited by 1,128
2023FrugalGPT: How to Use Large Language Models While Reducing Cost and Improving Performancecited by 853
2017Deep Reinforcement Learning that Matterscited by 754

5 more

2019Green AIcited by 627
2023API-Bank: A Comprehensive Benchmark for Tool-Augmented LLMscited by 591
2023Plan-and-Solve Prompting: Improving Zero-Shot Chain-of-Thought Reasoning by Large Language Modelscited by 560
2020Towards the Systematic Reporting of the Energy and Carbon Footprints of Machine Learningcited by 249
2019MLPerf Training Benchmarkcited by 115

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way