# FinRetrieval: A Benchmark for Financial Data Retrieval by AI Agents

arXiv: https://arxiv.org/abs/2603.04403 · HTML: https://mlsanity.com/paper/2603.04403/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 18 of its 22 references that are in the map.
- First posted on arXiv 2 Jan 2026.

## Builds on (most cited first, 18 of 18)

- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [ReAct: Synergizing Reasoning and Acting in Language Models](https://mlsanity.com/paper/2210.03629/) (2022)
- [AgentBench: Evaluating LLMs as Agents](https://mlsanity.com/paper/2308.03688/) (2023)
- [API-Bank: A Comprehensive Benchmark for Tool-Augmented LLMs](https://mlsanity.com/paper/2304.08244/) (2023)
- [FinQA: A Dataset of Numerical Reasoning over Financial Data](https://mlsanity.com/paper/2109.00122/) (2021)
- [TAT-QA: A Question Answering Benchmark on a Hybrid of Tabular and Textual Content in Finance](https://mlsanity.com/paper/2105.07624/) (2021)
- [Augmented Language Models: a Survey](https://mlsanity.com/paper/2302.07842/) (2023)
- [FinanceBench: A New Benchmark for Financial Question Answering](https://mlsanity.com/paper/2311.11944/) (2023)
- [FinBen: A Holistic Financial Benchmark for Large Language Models](https://mlsanity.com/paper/2402.12659/) (2024)
- [StableToolBench: Towards Stable Large-Scale Benchmarking on Tool Learning of Large Language Models](https://mlsanity.com/paper/2403.07714/) (2024)
- [CRAG -- Comprehensive RAG Benchmark](https://mlsanity.com/paper/2406.04744/) (2024)
- [Finance Agent Benchmark: Benchmarking LLMs on Real-world Financial Research Tasks](https://mlsanity.com/paper/2508.00828/) (2025)
- [DocFinQA: A Long-Context Financial Reasoning Dataset](https://mlsanity.com/paper/2401.06915/) (2024)
- [FinTextQA: A Dataset for Long-form Financial Question Answering](https://mlsanity.com/paper/2405.09980/) (2024)
- [FinAgentBench: A Benchmark Dataset for Agentic Retrieval in Financial Question Answering](https://mlsanity.com/paper/2508.14052/) (2025)
- [FAMMA: A Benchmark for Financial Domain Multilingual Multimodal Question Answering](https://mlsanity.com/paper/2410.04526/) (2024)
- [FinDER: Financial Dataset for Question Answering and Evaluating Retrieval-Augmented Generation](https://mlsanity.com/paper/2504.15800/) (2025)
- [Redefining Information Retrieval of Structured Database via Large Language Models](https://mlsanity.com/paper/2405.05508/) (2024)

## Built on by (most cited first, 4 of 4)

- [BigFinanceBench: A Workflow-Grounded Benchmark for Financial-Research Agents](https://mlsanity.com/paper/2606.03829/) (2026)
- [IPO Finance Agent: Benchmark of LLM Financial Analysts Beyond Finance Agent v2, with Automated Rubric Generation, on the SpaceX (SPCX) IPO](https://mlsanity.com/paper/2606.23032/) (2026)
- [FinRCA-Bench: Benchmarking Evidence Retrieval and Reasoning for Financial AI Systems](https://mlsanity.com/paper/2608.18534/) (2026)
- [FinAutoRubric: Expert-Guided Automatic Rubric Generation for Evaluating Financial Research Agents](https://mlsanity.com/paper/2609.35744/) (2026)

## Abstract (arXiv)

AI agents increasingly assist with financial research, yet no benchmark evaluates their ability to retrieve specific numeric values from structured databases. We introduce FinRetrieval, a benchmark of 500 financial retrieval questions with ground truth answers, agent responses from 14 configurations across three frontier providers (Anthropic, OpenAI, Google), and complete tool call execution traces. Our evaluation reveals that tool availability dominates performance: Claude Opus achieves 90.8% accuracy with structured data APIs but only 19.8% with web search alone--a 71 percentage point gap that exceeds other providers by 3-4x. We find that reasoning mode benefits vary inversely with base capability (+9.0pp for OpenAI vs +2.8pp for Claude), explained by differences in base-mode tool utilization rather than reasoning ability. Geographic performance gaps (5.6pp US advantage) stem from fiscal year naming conventions, not model limitations. We release the dataset, evaluation code, and tool traces to enable research on financial AI systems.