# Finance Agent Benchmark: Benchmarking LLMs on Real-world Financial Research Tasks

arXiv: https://arxiv.org/abs/2508.00828 · HTML: https://mlsanity.com/paper/2508.00828/

## Key facts

- Cited by 65 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 20 May 2025.

## Built on by (most cited first, 30 of 65)

- [Towards a Science of Scaling Agent Systems](https://mlsanity.com/paper/2512.08296/) (2025)
- [StockBench: Can LLM Agents Trade Stocks Profitably In Real-world Markets?](https://mlsanity.com/paper/2510.02209/) (2025)
- [Nemotron 3 Ultra: Open, Efficient Mixture-of-Experts Hybrid Mamba-Transformer Model for Agentic Reasoning](https://mlsanity.com/paper/2606.15007/) (2026)
- [FinSearchComp: Towards a Realistic, Expert-Level Evaluation of Financial Search and Reasoning](https://mlsanity.com/paper/2509.13160/) (2025)
- [Uncertainty Quantification in LLM Agents: Foundations, Emerging Challenges, and Opportunities](https://mlsanity.com/paper/2602.05073/) (2026)
- [Continual Learning Bench: Evaluating Frontier AI Systems in Real-World Stateful Environments](https://mlsanity.com/paper/2606.05661/) (2026)
- [Reinforcement Learning Foundations for Deep Research Systems: A Survey](https://mlsanity.com/paper/2509.06733/) (2025)
- [DRACO: a Cross-Domain Benchmark for Deep Research Accuracy, Completeness, and Objectivity](https://mlsanity.com/paper/2602.11685/) (2026)
- [LiveTradeBench: Seeking Real-World Alpha with Large Language Models](https://mlsanity.com/paper/2511.03628/) (2025)
- [Automated Benchmark Auditing for AI Agents and Large Language Models](https://mlsanity.com/paper/2605.26079/) (2026)
- [ToolFailBench: Diagnosing Tool-Use Failures in LLM Agents](https://mlsanity.com/paper/2607.04686/) (2026)
- [Finch: Benchmarking Finance & Accounting across Spreadsheet-Centric Enterprise Workflows](https://mlsanity.com/paper/2512.13168/) (2025)
- [FinToolBench: Evaluating LLM Agents for Real-World Financial Tool Use](https://mlsanity.com/paper/2603.08262/) (2026)
- [Agentic Environment Engineering for Large Language Models: A Survey of Environment Modeling, Synthesis, Evaluation, and Application](https://mlsanity.com/paper/2606.12191/) (2026)
- [BigFinanceBench: A Workflow-Grounded Benchmark for Financial-Research Agents](https://mlsanity.com/paper/2606.03829/) (2026)
- [CryptoBench: A Dynamic Benchmark for Expert-Level Evaluation of LLM Agents in Cryptocurrency](https://mlsanity.com/paper/2512.00417/) (2025)
- [TacoMAS: Test-Time Co-Evolution of Topology and Capability in LLM-based Multi-Agent Systems](https://mlsanity.com/paper/2605.09539/) (2026)
- [Economy of Minds: Emerging Multi-Agent Intelligence with Economic Interactions](https://mlsanity.com/paper/2606.02859/) (2026)
- [VERAFI: Verified Agentic Financial Intelligence through Neurosymbolic Policy Generation](https://mlsanity.com/paper/2512.14744/) (2025)
- [TraderBench: How Robust Are AI Agents in Adversarial Capital Markets?](https://mlsanity.com/paper/2603.00285/) (2026)
- [FinRetrieval: A Benchmark for Financial Data Retrieval by AI Agents](https://mlsanity.com/paper/2603.04403/) (2026)
- [Can LLM Agents Be CFOs? Benchmarking Long-Horizon Resource Allocation in an Uncertain Enterprise Environment](https://mlsanity.com/paper/2603.23638/) (2026)
- [QuantCode-Bench: A Benchmark for Evaluating the Ability of Large Language Models to Generate Executable Algorithmic Trading Strategies](https://mlsanity.com/paper/2604.15151/) (2026)
- [Herculean: An Agentic Benchmark for Financial Intelligence](https://mlsanity.com/paper/2605.14355/) (2026)
- [AgentStream: How Well Do Self-Evolving LLM Agents Perform Under Streaming Tasks?](https://mlsanity.com/paper/2608.00155/) (2026)
- [FrontierFinance: A Challenging Benchmark for Measuring Frontier Intelligence of Finance Agents](https://mlsanity.com/paper/2608.11683/) (2026)
- [When Does Multi-Agent Collaboration Help? An Entropy Perspective](https://mlsanity.com/paper/2602.04234/) (2026)
- [The Price of Agreement: Measuring LLM Sycophancy in Agentic Financial Applications](https://mlsanity.com/paper/2604.24668/) (2026)
- [FinEvo-Bench: A Longitudinal Benchmark for Self-Evolving Agents in Professional Financial Workflows](https://mlsanity.com/paper/2608.06144/) (2026)
- [BankerToolBench: Evaluating AI Agents in End-to-End Investment Banking Workflows](https://mlsanity.com/paper/2604.11304/) (2026)

## Abstract (arXiv)

Artificial Intelligence (AI) technology has emerged as a transformative force in financial analysis and the finance industry, though significant questions remain about the full capabilities of Large Language Model (LLM) agents in this domain. We present the Finance Agent Benchmark, featuring challenging and diverse real-world finance research problems that require LLMs to perform complex analysis using recent SEC filings. We construct the benchmark using a taxonomy of nine financial task categories, developed in consultation with experts from banks, hedge funds, and private equity firms. The dataset includes 537 expert-authored questions covering tasks from information retrieval to complex financial modeling, each validated through a rigorous review process to ensure accuracy and relevance. Moreover, we implement an agentic harness that equips LLMs with tools sufficient to produce accurate responses, including Google Search and EDGAR database access. Overall, the Finance Agent Benchmark provides a comprehensive testbed for measuring the progress of LLM-driven finance agents. Our evaluation reveals significant limitations in current AI capabilities - even the best-performing model (OpenAI o3) achieved only 46.8% accuracy at an average cost of 3.79 per query. This underscores the need for further advancements before reliable deployment in high-stakes finance settings.