# AgentArcEval: An Architecture Evaluation Method for Foundation Model based Agents

arXiv: https://arxiv.org/abs/2510.21031 · HTML: https://mlsanity.com/paper/2510.21031/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Placed in General LLM agent frameworks and benchmarks, one of 16 directions in LLM agents and tool use.
- Builds on 21 of its 66 references that are in the map.
- First posted on arXiv 23 Oct 2025.

## Placed in

- LLM agents and tool use / [General LLM agent frameworks and benchmarks](https://mlsanity.com/agents/general-llm-agent-frameworks-and-benchmarks/)

## Builds on (most cited first, 21 of 21)

- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [ReAct: Synergizing Reasoning and Acting in Language Models](https://mlsanity.com/paper/2210.03629/) (2022)
- [On the Opportunities and Risks of Foundation Models](https://mlsanity.com/paper/2108.07258/) (2021)
- [Generative Agents: Interactive Simulacra of Human Behavior](https://mlsanity.com/paper/2304.03442/) (2023)
- [Self-Refine: Iterative Refinement with Self-Feedback](https://mlsanity.com/paper/2303.17651/) (2023)
- [Tree of Thoughts: Deliberate Problem Solving with Large Language Models](https://mlsanity.com/paper/2305.10601/) (2023)
- [Voyager: An Open-Ended Embodied Agent with Large Language Models](https://mlsanity.com/paper/2305.16291/) (2023)
- [MemGPT: Towards LLMs as Operating Systems](https://mlsanity.com/paper/2310.08560/) (2023)
- [Inner Monologue: Embodied Reasoning through Planning with Language Models](https://mlsanity.com/paper/2207.05608/) (2022)
- [ExpeL: LLM Agents Are Experiential Learners](https://mlsanity.com/paper/2308.10144/) (2023)
- [Multi-Agent Collaboration: Harnessing the Power of Intelligent LLM Agents](https://mlsanity.com/paper/2306.03314/) (2023)
- [Building Cooperative Embodied Agents Modularly with Large Language Models](https://mlsanity.com/paper/2307.02485/) (2023)
- [AutoAgents: A Framework for Automatic Agent Generation](https://mlsanity.com/paper/2309.17288/) (2023)
- [OpenAgents: An Open Platform for Language Agents in the Wild](https://mlsanity.com/paper/2310.10634/) (2023)
- [TPTU-v2: Boosting Task Planning and Tool Usage of Large Language Model-based Agents in Real-world Systems](https://mlsanity.com/paper/2311.11315/) (2023)
- [Open-Ended Instructable Embodied Agents with Memory-Augmented Large Language Models](https://mlsanity.com/paper/2310.15127/) (2023)
- [Self-Adaptive Large Language Model (LLM)-Based Multiagent Systems](https://mlsanity.com/paper/2307.06187/) (2023)
- [InterAct: Exploring the Potentials of ChatGPT as a Cooperative Agent](https://mlsanity.com/paper/2308.01552/) (2023)
- [Towards Responsible Generative AI: A Reference Architecture for Designing Foundation Model based Agents](https://mlsanity.com/paper/2311.13148/) (2023)
- [A Functional Software Reference Architecture for LLM-Integrated Systems](https://mlsanity.com/paper/2501.12904/) (2025)
- [SeeHow: Workflow Extraction from Programming Screencasts through Action-Aware Video Analytics](https://mlsanity.com/paper/2304.14042/) (2023)

## Built on by (most cited first, 2 of 2)

- [A Reference Architecture for Agentic Hybrid Retrieval in Dataset Search](https://mlsanity.com/paper/2604.16394/) (2026)
- [NostrAgent: A Decentralized Identity and Delegation Architecture for Sovereign Agentic Systems](https://mlsanity.com/paper/2609.22944/) (2026)

## Abstract (arXiv)

The emergence of foundation models (FMs) has enabled the development of highly capable and autonomous agents, unlocking new application opportunities across a wide range of domains. Evaluating the architecture of agents is particularly important as the architectural decisions significantly impact the quality attributes of agents given their unique characteristics, including compound architecture, autonomous and non-deterministic behaviour, and continuous evolution. However, these traditional methods fall short in addressing the evaluation needs of agent architecture due to the unique characteristics of these agents. Therefore, in this paper, we present AgentArcEval, a novel agent architecture evaluation method designed specially to address the complexities of FM-based agent architecture and its evaluation. Moreover, we present a catalogue of agent-specific general scenarios, which serves as a guide for generating concrete scenarios to design and evaluate the agent architecture. We demonstrate the usefulness of AgentArcEval and the catalogue through a case study on the architecture evaluation of a real-world tax copilot, named Luna.