Benchmarking Multi-Agent LLM Architectures for Financial Document Processing: A Comparative Study of Orchestration Patterns, Cost-Accuracy Tradeoffs and Production Scaling Strategies
On the map: LLM agents and tool use → Financial trading and forecasting agents
Abstract (arXiv)
The adoption of large language models (LLMs) for structured information extraction from financial documents has accelerated rapidly, yet production deployments face fundamental architectural decisions with limited empirical guidance. We present a systematic benchmark comparing four multi-agent orchestration architectures: sequential pipeline, parallel fan-out with merge, hierarchical supervisor-worker and reflexive self-correcting loop. These are evaluated across five frontier and open-weight LLMs on a corpus of 10,000 SEC filings (10-K, 10-Q and 8-K forms). Our evaluation spans 25 extraction field types covering governance structures, executive compensation and financial metrics, measured along five axes: field-level F1, document-level accuracy, end-to-end latency, cost per document and token efficiency. We find that reflexive architectures achieve the highest field-level F1 (0.943) but at 2.3x the cost of sequential baselines, while hierarchical architectures occupy the most favorable position on the cost-accuracy Pareto frontier (F1 0.921 at 1.4x cost). We further present ablation studies on semantic caching, model routing and adaptive retry strategies, demonstrating that hybrid configurations can recover 89% of the reflexive architecture's accuracy gains at only 1.15x baseline cost. Our scaling analysis from 1K to 100K documents per day reveals non-obvious throughput-accuracy degradation curves that inform capacity planning. These findings provide actionable guidance for practitioners deploying multi-agent LLM systems in regulated financial environments.
Key facts. Cited by 6 papers in the map (data through 7 Oct 2026). Placed in Financial trading and forecasting agents, one of 16 directions in LLM agents and tool use. Builds on 15 of its 30 references that are in the map. First posted on arXiv 24 Mar 2026.
Builds on
15 of 30 references.
References · 15
- 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,355
- 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
- 2023Reflexion: Language Agents with Verbal Reinforcement Learningcited by 4,064
- 2023Generative Agents: Interactive Simulacra of Human Behaviorcited by 3,500
- 2023Self-Refine: Iterative Refinement with Self-Feedbackcited by 3,317
- 2023AutoGen: Enabling Next-Gen LLM Applications via Multi-Agent Conversationcited by 1,906
- 2023MetaGPT: Meta Programming for A Multi-Agent Collaborative Frameworkcited by 1,761
- 2023A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questionscited by 1,582
- 2023BloombergGPT: A Large Language Model for Financecited by 923
- 2023Ragas: Automated Evaluation of Retrieval Augmented Generationcited by 408
- 2023How is ChatGPT's behavior changing over time?cited by 265
- 2024FinBen: A Holistic Financial Benchmark for Large Language Modelscited by 140
Built on by
Citing papers per year: 2026: 6.
6 papers.
- 2026MADP: A Multi-Agent Pipeline for Sustainable Document Processing with Human-in-the-Loopcited by 1
- 2026The Path to Self-Evolving Clinical Systems: Scaling Medical Agents from Assistance to Autonomycited by 1
- 2026OrchestraBench: Evaluating Multi-Agent Orchestration Failure Modes, Recovery, and Decomposition Qualitycited by 1
- 2026Capital Markets LLM Reliability Score (CM-LRS): From Plausible to Bankablecited by 0
- 2026Bridging Search and CRM: Productionizing AI Product Research Agents for Customer Re-Engagementcited by 0
- 2026Beyond Accuracy: Robustness, Cost, and Governance Trade-offs for Vision-Language Models in Templated Document Extractioncited by 0
- 2026Beyond Accuracy: Robustness, Cost, and Governance Trade-offs for Vision-Language Models in Templated Document Extraction14 Sep 2026
- 2026Bridging Search and CRM: Productionizing AI Product Research Agents for Customer Re-Engagement19 Aug 2026
- 2026OrchestraBench: Evaluating Multi-Agent Orchestration Failure Modes, Recovery, and Decomposition Quality5 Aug 2026
- 2026Capital Markets LLM Reliability Score (CM-LRS): From Plausible to Bankable23 Jul 2026
- 2026The Path to Self-Evolving Clinical Systems: Scaling Medical Agents from Assistance to Autonomy13 Jul 2026
- 2026MADP: A Multi-Agent Pipeline for Sustainable Document Processing with Human-in-the-Loop16 May 2026
Similar, not cited
Close in wording, no citation either way
- 2026BlueFin: Benchmarking LLM Agents on Financial Spreadsheets
- 2026FinToolBench: Evaluating LLM Agents for Real-World Financial Tool Use
- 2026OmegaUse-OfficeVal: Benchmarking LLM Agents on Long-Horizon Office-Suite Tasks with Economic Grounding
- 2026FinMCP-Bench: Benchmarking LLM Agents for Real-World Financial Tool Use under the Model Context Protocol
- 2025On the Comprehensibility of Multi-structured Financial Documents using LLMs and Pre-processing Tools ↗
- 2025Structuring the Unstructured: A Multi-Agent System for Extracting and Querying Financial KPIs and Guidance
- 2026Time Series Augmented Generation for Financial Applications
- 2026FinReasoning: A Hierarchical Benchmark for Reliable Financial Research Reporting