# AgentArch: A Comprehensive Benchmark to Evaluate Agent Architectures in Enterprise

arXiv: https://arxiv.org/abs/2509.10769 · HTML: https://mlsanity.com/paper/2509.10769/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Placed in General LLM agent frameworks and benchmarks, one of 16 directions in LLM agents and tool use.
- Builds on 16 of its 25 references that are in the map.
- First posted on arXiv 13 Sep 2025.

## Placed in

- LLM agents and tool use / [General LLM agent frameworks and benchmarks](https://mlsanity.com/agents/general-llm-agent-frameworks-and-benchmarks/)

## Builds on (most cited first, 16 of 16)

- [ReAct: Synergizing Reasoning and Acting in Language Models](https://mlsanity.com/paper/2210.03629/) (2022)
- [τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains](https://mlsanity.com/paper/2406.12045/) (2024)
- [MemGPT: Towards LLMs as Operating Systems](https://mlsanity.com/paper/2310.08560/) (2023)
- [A-MEM: Agentic Memory for LLM Agents](https://mlsanity.com/paper/2502.12110/) (2025)
- [Evaluating Very Long-Term Conversational Memory of LLM Agents](https://mlsanity.com/paper/2402.17753/) (2024)
- [WorkArena: How Capable Are Web Agents at Solving Common Knowledge Work Tasks?](https://mlsanity.com/paper/2403.07718/) (2024)
- [MultiAgentBench: Evaluating the Collaboration and Competition of LLM agents](https://mlsanity.com/paper/2503.01935/) (2025)
- [EvoAgent: Towards Automatic Multi-Agent Generation via Evolutionary Algorithms](https://mlsanity.com/paper/2406.14228/) (2024)
- [CRMArena: Understanding the Capacity of LLM Agents to Perform Professional CRM Tasks in Realistic Environments](https://mlsanity.com/paper/2411.02305/) (2024)
- [LLM-Coordination: Evaluating and Analyzing Multi-agent Coordination Abilities in Large Language Models](https://mlsanity.com/paper/2310.03903/) (2023)
- [Seal-Tools: Self-Instruct Tool Learning Dataset for Agent Tuning and Detailed Benchmark](https://mlsanity.com/paper/2405.08355/) (2024)
- [WorkBench: a Benchmark Dataset for Agents in a Realistic Workplace Setting](https://mlsanity.com/paper/2405.00823/) (2024)
- [NESTFUL: A Benchmark for Evaluating LLMs on Nested Sequences of API Calls](https://mlsanity.com/paper/2409.03797/) (2024)
- [CRMArena-Pro: Holistic Assessment of LLM Agents Across Diverse Business Scenarios and Interactions](https://mlsanity.com/paper/2505.18878/) (2025)
- [X-MAS: Towards Building Multi-Agent Systems with Heterogeneous LLMs](https://mlsanity.com/paper/2505.16997/) (2025)
- [HALO: Hierarchical Autonomous Logic-Oriented Orchestration for Multi-Agent LLM Systems](https://mlsanity.com/paper/2505.13516/) (2025)

## Built on by (most cited first, 8 of 8)

- [PAST-Bench: Benchmarking the Foundations of Recursive Self-Improvement in Personal Agents](https://mlsanity.com/paper/2608.04003/) (2026)
- [An Empirical Study of Harness Design for Coding Agents](https://mlsanity.com/paper/2609.20804/) (2026)
- [KAIJU: An Executive Kernel for Intent-Gated Execution of LLM Agents](https://mlsanity.com/paper/2604.02375/) (2026)
- [Profile-Then-Reason: Bounded Semantic Complexity for Tool-Augmented Language Agents](https://mlsanity.com/paper/2604.04131/) (2026)
- [When Cloud Agents Meet Device Agents: Lessons from Hybrid Multi-Agent Systems](https://mlsanity.com/paper/2605.30102/) (2026)
- [Benchmarking AI Agents for Hardware Design Automation via MCP Tool Calling](https://mlsanity.com/paper/2608.26199/) (2026)
- [ChurnBench: A Drift-Aware Benchmark Demonstrating That Refresh Scheduling, Not Cache Age, Governs Staleness in Agentic AI](https://mlsanity.com/paper/2609.11515/) (2026)
- [Continuous Process-Level Evaluation for Evolving Enterprise AI Agent Skills](https://mlsanity.com/paper/2610.01833/) (2026)

## Abstract (arXiv)

While individual components of agentic architectures have been studied in isolation, there remains limited empirical understanding of how different design dimensions interact within complex multi-agent systems. This study aims to address these gaps by providing a comprehensive enterprise-specific benchmark evaluating 18 distinct agentic configurations across state-of-the-art large language models. We examine four critical agentic system dimensions: orchestration strategy, agent prompt implementation (ReAct versus function calling), memory architecture, and thinking tool integration. Our benchmark reveals significant model-specific architectural preferences that challenge the prevalent one-size-fits-all paradigm in agentic AI systems. It also reveals significant weaknesses in overall agentic performance on enterprise tasks with the highest scoring models achieving a maximum of only 35.3% success on the more complex task and 70.8% on the simpler task. We hope these findings inform the design of future agentic systems by enabling more empirically backed decisions regarding architectural components and model selection.