# FinAgentBench: A Benchmark Dataset for Agentic Retrieval in Financial Question Answering

arXiv: https://arxiv.org/abs/2508.14052 · HTML: https://mlsanity.com/paper/2508.14052/

## Key facts

- Cited by 29 papers in the map (data through 14 Oct 2026).
- Placed in Financial trading and forecasting agents, one of 16 directions in LLM agents and tool use.
- Builds on 13 of its 26 references that are in the map.
- First posted on arXiv 7 Aug 2025.

## Placed in

- LLM agents and tool use / [Financial trading and forecasting agents](https://mlsanity.com/agents/financial-trading-and-forecasting-agents/)

## Builds on (most cited first, 13 of 13)

- [ReAct: Synergizing Reasoning and Acting in Language Models](https://mlsanity.com/paper/2210.03629/) (2022)
- [Dense Passage Retrieval for Open-Domain Question Answering](https://mlsanity.com/paper/2004.04906/) (2020)
- [FinQA: A Dataset of Numerical Reasoning over Financial Data](https://mlsanity.com/paper/2109.00122/) (2021)
- [TAT-QA: A Question Answering Benchmark on a Hybrid of Tabular and Textual Content in Finance](https://mlsanity.com/paper/2105.07624/) (2021)
- [FinanceBench: A New Benchmark for Financial Question Answering](https://mlsanity.com/paper/2311.11944/) (2023)
- [DocFinQA: A Long-Context Financial Reasoning Dataset](https://mlsanity.com/paper/2401.06915/) (2024)
- [How Does Generative Retrieval Scale to Millions of Passages?](https://mlsanity.com/paper/2305.11841/) (2023)
- [Don't Do RAG: When Cache-Augmented Generation is All You Need for Knowledge Tasks](https://mlsanity.com/paper/2412.15605/) (2024)
- [Large Language Models as Foundations for Next-Gen Dense Retrieval: A Comprehensive Empirical Assessment](https://mlsanity.com/paper/2408.12194/) (2024)
- [Robust Neural Information Retrieval: An Adversarial and Out-of-distribution Perspective](https://mlsanity.com/paper/2407.06992/) (2024)
- [Infinite Retrieval: Attention Enhanced LLMs in Long-Context Processing](https://mlsanity.com/paper/2502.12962/) (2025)
- [Hierarchical Retrieval with Evidence Curation for Open-Domain Financial Question Answering on Standardized Documents](https://mlsanity.com/paper/2505.20368/) (2025)
- [Exploring Training and Inference Scaling Laws in Generative Retrieval](https://mlsanity.com/paper/2503.18941/) (2025)

## Built on by (most cited first, 29 of 29)

- [A Comprehensive Survey on Reinforcement Learning-based Agentic Search: Foundations, Roles, Optimizations, Evaluations, and Applications](https://mlsanity.com/paper/2510.16724/) (2025)
- [FinCARDS: Card-Based Analyst Reranking for Financial Document Question Answering](https://mlsanity.com/paper/2601.06992/) (2026)
- [DIVE: Scaling Diversity in Agentic Task Synthesis for Generalizable Tool Use](https://mlsanity.com/paper/2603.11076/) (2026)
- [Evaluating LLMs in Finance Requires Explicit Bias Consideration](https://mlsanity.com/paper/2602.14233/) (2026)
- [Finch: Benchmarking Finance & Accounting across Spreadsheet-Centric Enterprise Workflows](https://mlsanity.com/paper/2512.13168/) (2025)
- [BigFinanceBench: A Workflow-Grounded Benchmark for Financial-Research Agents](https://mlsanity.com/paper/2606.03829/) (2026)
- [FinRetrieval: A Benchmark for Financial Data Retrieval by AI Agents](https://mlsanity.com/paper/2603.04403/) (2026)
- [FinTrace: Holistic Trajectory-Level Evaluation of LLM Tool Calling for Long-Horizon Financial Tasks](https://mlsanity.com/paper/2604.10015/) (2026)
- [QuantCode-Bench: A Benchmark for Evaluating the Ability of Large Language Models to Generate Executable Algorithmic Trading Strategies](https://mlsanity.com/paper/2604.15151/) (2026)
- [Herculean: An Agentic Benchmark for Financial Intelligence](https://mlsanity.com/paper/2605.14355/) (2026)
- [FrontierFinance: A Challenging Benchmark for Measuring Frontier Intelligence of Finance Agents](https://mlsanity.com/paper/2608.11683/) (2026)
- [FinEvo-Bench: A Longitudinal Benchmark for Self-Evolving Agents in Professional Financial Workflows](https://mlsanity.com/paper/2608.06144/) (2026)
- [Navigating Large-Scale Document Collections: MuDABench for Multi-Document Analytical QA](https://mlsanity.com/paper/2604.22239/) (2026)
- [Point-in-Time Financial RAG with Frozen LLMs and Market-Feedback Adaptive Retrieval](https://mlsanity.com/paper/2605.31201/) (2026)
- [FORCE-Bench: A Benchmark, Dataset, and Evaluation Harness for Agentic AI in Enterprise Finance](https://mlsanity.com/paper/2607.19409/) (2026)
- [Streaming Knowledge Compilation: Proactive Materiality-Scored Pinning for Time-Evolving LLM Wikis](https://mlsanity.com/paper/2606.09877/) (2026)
- [Understanding the Behaviors of Environment-aware Information Retrieval](https://mlsanity.com/paper/2606.16817/) (2026)
- [Bringing Agentic Search to Earth Observation Data Discovery](https://mlsanity.com/paper/2607.02387/) (2026)
- [DFAH-Bench: Benchmarking Observable Agent Instability in Financial Decision-Making](https://mlsanity.com/paper/2607.20491/) (2026)
- [HVM-GraphRAG: Holistic-View Multimodal Graph Retrieval-Augmented Generation on Complex Document](https://mlsanity.com/paper/2607.24861/) (2026)
- [FinDeepIndicator: Benchmarking Deep Research Agents in End-to-End Financial Indicator Construction](https://mlsanity.com/paper/2608.00764/) (2026)
- [FinRank: An Evidence-Grounded Benchmark for Financial Question Answering and Retrieval over SEC Filings](https://mlsanity.com/paper/2608.07400/) (2026)
- [EvolveTrade: Experience-Driven Policy Refinement for Self-Evolving LLM Trading Agents](https://mlsanity.com/paper/2609.17632/) (2026)
- [Absorbing Complexity: An Interaction-Native Knowledge Harness for Financial LLM Agents](https://mlsanity.com/paper/2606.01886/) (2026)
- [Frontier Financial Judgement: Can agents tell what might move a stock?](https://mlsanity.com/paper/2607.20645/) (2026)
- [Seeing Is Not Deciding: Can Multimodal LLMs Act as Effective CEOs?](https://mlsanity.com/paper/2608.05864/) (2026)
- [FinRCA-Bench: Benchmarking Evidence Retrieval and Reasoning for Financial AI Systems](https://mlsanity.com/paper/2608.18534/) (2026)
- [FinInteract: Benchmarking Clarification and Intent Integration in Ambiguous Financial Question Answering](https://mlsanity.com/paper/2609.24002/) (2026)
- [FinAutoRubric: Expert-Guided Automatic Rubric Generation for Evaluating Financial Research Agents](https://mlsanity.com/paper/2609.35744/) (2026)

## Abstract (arXiv)

Accurate information retrieval (IR) is critical in the financial domain, where investors must identify relevant information from large collections of documents. Traditional IR methods -- whether sparse or dense -- often fall short in retrieval accuracy, as it requires not only capturing semantic similarity but also performing fine-grained reasoning over document structure and domain-specific knowledge. Recent advances in large language models (LLMs) have opened up new opportunities for retrieval with multi-step reasoning, where the model ranks passages through iterative reasoning about which information is most relevant to a given query. However, there exists no benchmark to evaluate such capabilities in the financial domain. To address this gap, we introduce FinAgentBench, the first large-scale benchmark for evaluating retrieval with multi-step reasoning in finance -- a setting we term agentic retrieval. The benchmark consists of 26K expert-annotated examples on S&P-500 listed firms and assesses whether LLM agents can (1) identify the most relevant document type among candidates, and (2) pinpoint the key passage within the selected document. Our evaluation framework explicitly separates these two reasoning steps to address context limitations. This design enables to provide a quantitative basis for understanding retrieval-centric LLM behavior in finance. We evaluate a suite of state-of-the-art models and further demonstrated how targeted fine-tuning can significantly improve agentic retrieval performance. Our benchmark provides a foundation for studying retrieval-centric LLM behavior in complex, domain-specific tasks for finance.