# Toward Reliable Evaluation of LLM-Based Financial Multi-Agent Systems: Taxonomy, Coordination Primacy, and Cost Awareness

arXiv: https://arxiv.org/abs/2603.27539 · HTML: https://mlsanity.com/paper/2603.27539/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Placed in Financial trading and forecasting agents, one of 16 directions in LLM agents and tool use.
- Builds on 21 of its 29 references that are in the map.
- First posted on arXiv 29 Mar 2026.

## Placed in

- LLM agents and tool use / [Financial trading and forecasting agents](https://mlsanity.com/agents/financial-trading-and-forecasting-agents/)

## Builds on (most cited first, 21 of 21)

- [Improving Factuality and Reasoning in Language Models through Multiagent Debate](https://mlsanity.com/paper/2305.14325/) (2023)
- [Large Language Model based Multi-Agents: A Survey of Progress and Challenges](https://mlsanity.com/paper/2402.01680/) (2024)
- [Multi-Agent Collaboration: Harnessing the Power of Intelligent LLM Agents](https://mlsanity.com/paper/2306.03314/) (2023)
- [Small Language Models are the Future of Agentic AI](https://mlsanity.com/paper/2506.02153/) (2025)
- [TradingAgents: Multi-Agents LLM Financial Trading Framework](https://mlsanity.com/paper/2412.20138/) (2024)
- [FinMem: A Performance-Enhanced LLM Trading Agent with Layered Memory and Character Design](https://mlsanity.com/paper/2311.13743/) (2023)
- [A Multimodal Foundation Agent for Financial Trading: Tool-Augmented, Diversified, and Generalist](https://mlsanity.com/paper/2402.18485/) (2024)
- [FinCon: A Synthesized LLM Multi-Agent System with Conceptual Verbal Reinforcement for Enhanced Financial Decision Making](https://mlsanity.com/paper/2407.06567/) (2024)
- [How Memory Management Impacts LLM Agents: An Empirical Study of Experience-Following Behavior](https://mlsanity.com/paper/2505.16067/) (2025)
- [Debate or Vote: Which Yields Better Decisions in Multi-Agent Large Language Models?](https://mlsanity.com/paper/2508.17536/) (2025)
- [LLM-based Multi-Agent Reinforcement Learning: Current and Future Directions](https://mlsanity.com/paper/2405.11106/) (2024)
- [Large Language Model Agent in Financial Trading: A Survey](https://mlsanity.com/paper/2408.06361/) (2024)
- [StockBench: Can LLM Agents Trade Stocks Profitably In Real-world Markets?](https://mlsanity.com/paper/2510.02209/) (2025)
- [FinRobot: An Open-Source AI Agent Platform for Financial Applications using Large Language Models](https://mlsanity.com/paper/2405.14767/) (2024)
- [HedgeAgents: A Balanced-aware Multi-agent Financial Trading System](https://mlsanity.com/paper/2502.13165/) (2025)
- [Can LLM-based Financial Investing Strategies Outperform the Market in Long Run?](https://mlsanity.com/paper/2505.07078/) (2025)
- [When Agents Trade: Live Multi-Market Trading Benchmark for LLM Agents](https://mlsanity.com/paper/2510.11695/) (2025)
- [QuantAgents: Towards Multi-agent Financial System via Simulated Trading](https://mlsanity.com/paper/2510.04643/) (2025)
- [FinVision: A Multi-Agent Framework for Stock Market Prediction](https://mlsanity.com/paper/2411.08899/) (2024)
- [Integrating Traditional Technical Analysis with AI: A Multi-Agent LLM-Based Approach to Stock Market Forecasting](https://mlsanity.com/paper/2506.16813/) (2025)
- [Design and Empirical Study of a Large Language Model-Based Multi-Agent Investment System for Chinese Public REITs](https://mlsanity.com/paper/2602.00082/) (2026)

## Built on by (most cited first, 5 of 5)

- [Agentic Trading: When LLM Agents Meet Financial Markets](https://mlsanity.com/paper/2605.19337/) (2026)
- [Symbolic-Vector Attention Fusion for Collective Intelligence](https://mlsanity.com/paper/2604.03955/) (2026)
- [Look-Ahead-Freedom as Temporal Non-Interference: A Verifiable Correctness Property for Backtesting and Agentic Trading Pipelines](https://mlsanity.com/paper/2607.04958/) (2026)
- [CLAIR-Fin: An Adversarial Multi-Agent Framework for Claim-Level Verification and Adaptive Debate in Cross-Modal Financial QA](https://mlsanity.com/paper/2608.13706/) (2026)
- [DynaContext: Self-Improving Dynamic Contextualization of Optimized Prompts for Heterogeneous Parameter Extraction](https://mlsanity.com/paper/2608.22014/) (2026)

## Abstract (arXiv)

Multi-agent systems based on large language models (LLMs) for financial trading have grown rapidly since 2023, yet the field lacks a shared framework for understanding what drives performance or for evaluating claims credibly. This survey makes three contributions. First, we introduce a four-dimensional taxonomy, covering architecture pattern, coordination mechanism, memory architecture, and tool integration; applied to 12 multi-agent systems and two single-agent baselines. Second, we formulate the Coordination Primacy Hypothesis (CPH): inter-agent coordination protocol design is a primary driver of trading decision quality, often exerting greater influence than model scaling. CPH is presented as a falsifiable research hypothesis supported by tiered structural evidence rather than as an empirically validated conclusion; its definitive validation requires evaluation infrastructure that does not yet exist in the field. Third, we document five pervasive evaluation failures (look-ahead bias, survivorship bias, backtesting overfitting, transaction cost neglect, and regime-shift blindness) and show that these can reverse the sign of reported returns. Building on the CPH and the evaluation critique, we introduce the Coordination Breakeven Spread (CBS), a metric for determining whether multi-agent coordination adds genuine value net of transaction costs, and propose minimum evaluation standards as prerequisites for validating the CPH.