mlsanityData to 7 Oct 2026

AgentArcEval: An Architecture Evaluation Method for Foundation Model based Agents

arXiv 2510.21031 ↗First posted 23 Oct 2025cs.SE cs.AIcited by 2

On the map: LLM agents and tool use → General LLM agent frameworks and benchmarks

Abstract (arXiv)

The emergence of foundation models (FMs) has enabled the development of highly capable and autonomous agents, unlocking new application opportunities across a wide range of domains. Evaluating the architecture of agents is particularly important as the architectural decisions significantly impact the quality attributes of agents given their unique characteristics, including compound architecture, autonomous and non-deterministic behaviour, and continuous evolution. However, these traditional methods fall short in addressing the evaluation needs of agent architecture due to the unique characteristics of these agents. Therefore, in this paper, we present AgentArcEval, a novel agent architecture evaluation method designed specially to address the complexities of FM-based agent architecture and its evaluation. Moreover, we present a catalogue of agent-specific general scenarios, which serves as a guide for generating concrete scenarios to design and evaluate the agent architecture. We demonstrate the usefulness of AgentArcEval and the catalogue through a case study on the architecture evaluation of a real-world tax copilot, named Luna.

Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Placed in General LLM agent frameworks and benchmarks, one of 16 directions in LLM agents and tool use. Builds on 21 of its 66 references that are in the map. First posted on arXiv 23 Oct 2025.

Builds on

21 of 66 references.

Closest ancestors · 8

Same direction, newest first

  1. 2023Towards Responsible Generative AI: A Reference Architecture for Designing Foundation Model based Agents22 Nov
  2. 2023TPTU-v2: Boosting Task Planning and Tool Usage of Large Language Model-based Agents in Real-world Systems19 Nov
  3. 2023OpenAgents: An Open Platform for Language Agents in the Wild16 Oct
  4. 2023AutoAgents: A Framework for Automatic Agent Generation29 Sep
  5. 2023ExpeL: LLM Agents Are Experiential Learners20 Aug
  6. 2023InterAct: Exploring the Potentials of ChatGPT as a Cooperative Agent3 Aug
  7. 2023Building Cooperative Embodied Agents Modularly with Large Language Models5 Jul
  8. 2023Multi-Agent Collaboration: Harnessing the Power of Intelligent LLM Agents5 Jun

Other references · 13

  1. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  2. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,355
  3. 2021On the Opportunities and Risks of Foundation Modelscited by 3,961
  4. 2023Generative Agents: Interactive Simulacra of Human Behaviorcited by 3,500
  5. 2023Self-Refine: Iterative Refinement with Self-Feedbackcited by 3,317
  6. 2023Tree of Thoughts: Deliberate Problem Solving with Large Language Modelscited by 3,155
  7. 2023Voyager: An Open-Ended Embodied Agent with Large Language Modelscited by 2,081
  8. 2023MemGPT: Towards LLMs as Operating Systemscited by 1,295
  9. 2022Inner Monologue: Embodied Reasoning through Planning with Language Modelscited by 1,057
  10. 2023Open-Ended Instructable Embodied Agents with Memory-Augmented Large Language Modelscited by 37
  11. 2023Self-Adaptive Large Language Model (LLM)-Based Multiagent Systemscited by 31
  12. 2025A Functional Software Reference Architecture for LLM-Integrated Systemscited by 8
1 more
  1. 2023SeeHow: Workflow Extraction from Programming Screencasts through Action-Aware Video Analyticscited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026A Reference Architecture for Agentic Hybrid Retrieval in Dataset Searchcited by 1
  2. 2026NostrAgent: A Decentralized Identity and Delegation Architecture for Sovereign Agentic Systemscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024A Taxonomy of Architecture Options for Foundation Model-based Agents: Analysis and Decision Model
  2. 2024Agent Design Pattern Catalogue: A Collection of Architectural Patterns for Foundation Model based Agents
  3. 2025AgentArch: A Comprehensive Benchmark to Evaluate Agent Architectures in Enterprise
  4. 2025An Empirical Study of Testing Practices in Open Source AI Agent Frameworks and Agentic Applications
  5. 2025OAgents: An Empirical Study of Building Effective Agents
  6. 2026MASEval: Extending Multi-Agent Evaluation from Models to Systems
  7. 2026Toward Architecture-Aware Evaluation Metrics for LLM Agents
  8. 2024Swiss Cheese Model for AI Safety: A Taxonomy and Reference Architecture for Multi-Layered Guardrails of Foundation Model Based Agents