mlsanityData to 7 Oct 2026

FinRetrieval: A Benchmark for Financial Data Retrieval by AI Agents

arXiv 2603.04403 ↗First posted 2 Jan 2026cs.IR cs.AI cs.CLcited by 4

Not on the map. Why?

Abstract (arXiv)

AI agents increasingly assist with financial research, yet no benchmark evaluates their ability to retrieve specific numeric values from structured databases. We introduce FinRetrieval, a benchmark of 500 financial retrieval questions with ground truth answers, agent responses from 14 configurations across three frontier providers (Anthropic, OpenAI, Google), and complete tool call execution traces. Our evaluation reveals that tool availability dominates performance: Claude Opus achieves 90.8% accuracy with structured data APIs but only 19.8% with web search alone--a 71 percentage point gap that exceeds other providers by 3-4x. We find that reasoning mode benefits vary inversely with base capability (+9.0pp for OpenAI vs +2.8pp for Claude), explained by differences in base-mode tool utilization rather than reasoning ability. Geographic performance gaps (5.6pp US advantage) stem from fiscal year naming conventions, not model limitations. We release the dataset, evaluation code, and tool traces to enable research on financial AI systems.

Key facts. Cited by 4 papers in the map (data through 7 Oct 2026). Builds on 18 of its 22 references that are in the map. First posted on arXiv 2 Jan 2026.

Builds on

18 of 22 references.

References · 18

  1. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  2. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,355
  3. 2023AgentBench: Evaluating LLMs as Agentscited by 1,211
  4. 2023API-Bank: A Comprehensive Benchmark for Tool-Augmented LLMscited by 574
  5. 2021FinQA: A Dataset of Numerical Reasoning over Financial Datacited by 552
  6. 2021TAT-QA: A Question Answering Benchmark on a Hybrid of Tabular and Textual Content in Financecited by 402
  7. 2023Augmented Language Models: a Surveycited by 398
  8. 2023FinanceBench: A New Benchmark for Financial Question Answeringcited by 261
  9. 2024FinBen: A Holistic Financial Benchmark for Large Language Modelscited by 140
  10. 2024StableToolBench: Towards Stable Large-Scale Benchmarking on Tool Learning of Large Language Modelscited by 129
  11. 2024CRAG -- Comprehensive RAG Benchmarkcited by 94
  12. 2025Finance Agent Benchmark: Benchmarking LLMs on Real-world Financial Research Taskscited by 63
6 more
  1. 2024DocFinQA: A Long-Context Financial Reasoning Datasetcited by 59
  2. 2024FinTextQA: A Dataset for Long-form Financial Question Answeringcited by 54
  3. 2025FinAgentBench: A Benchmark Dataset for Agentic Retrieval in Financial Question Answeringcited by 29
  4. 2024FAMMA: A Benchmark for Financial Domain Multilingual Multimodal Question Answeringcited by 19
  5. 2025FinDER: Financial Dataset for Question Answering and Evaluating Retrieval-Augmented Generationcited by 17
  6. 2024Redefining Information Retrieval of Structured Database via Large Language Modelscited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 4.

4 papers.

  1. 2026BigFinanceBench: A Workflow-Grounded Benchmark for Financial-Research Agentscited by 6
  2. 2026IPO Finance Agent: Benchmark of LLM Financial Analysts Beyond Finance Agent v2, with Automated Rubric Generation, on the SpaceX (SPCX) IPOcited by 0
  3. 2026FinRCA-Bench: Benchmarking Evidence Retrieval and Reasoning for Financial AI Systemscited by 0
  4. 2026FinAutoRubric: Expert-Guided Automatic Rubric Generation for Evaluating Financial Research Agentscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026FinFIRST: Benchmarking Search Agents for Financial Information Retrieval, Sourcing and Traceability
  2. 2026Hedge-Bench: Benchmarking Agents on Hard, Realistic Tasks Pertaining to Financial Reasoning
  3. 2025FinSearchComp: Towards a Realistic, Expert-Level Evaluation of Financial Search and Reasoning
  4. 2025Enhancing Financial RAG with Agentic AI and Multi-HyDE: A Novel Approach to Knowledge Retrieval and Hallucination Reduction
  5. 2026FinToolBench: Evaluating LLM Agents for Real-World Financial Tool Use
  6. 2026Agentic Retrieval-Augmented Generation for Financial Document Question Answering
  7. 2026FinInteract: Benchmarking Clarification and Intent Integration in Ambiguous Financial Question Answering
  8. 2026Herculean: An Agentic Benchmark for Financial Intelligence