# FinToolBench: Evaluating LLM Agents for Real-World Financial Tool Use

arXiv: https://arxiv.org/abs/2603.08262 · HTML: https://mlsanity.com/paper/2603.08262/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- Placed in Financial trading and forecasting agents, one of 16 directions in LLM agents and tool use.
- Builds on 24 of its 34 references that are in the map.
- First posted on arXiv 9 Mar 2026.

## Placed in

- LLM agents and tool use / [Financial trading and forecasting agents](https://mlsanity.com/agents/financial-trading-and-forecasting-agents/)

## Builds on (most cited first, 24 of 24)

- [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://mlsanity.com/paper/2306.05685/) (2023)
- [Toolformer: Language Models Can Teach Themselves to Use Tools](https://mlsanity.com/paper/2302.04761/) (2023)
- [G-Eval: NLG Evaluation using GPT-4 with Better Human Alignment](https://mlsanity.com/paper/2303.16634/) (2023)
- [WebArena: A Realistic Web Environment for Building Autonomous Agents](https://mlsanity.com/paper/2307.13854/) (2023)
- [ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIs](https://mlsanity.com/paper/2307.16789/) (2023)
- [τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains](https://mlsanity.com/paper/2406.12045/) (2024)
- [Gorilla: Large Language Model Connected with Massive APIs](https://mlsanity.com/paper/2305.15334/) (2023)
- [GAIA: a benchmark for General AI Assistants](https://mlsanity.com/paper/2311.12983/) (2023)
- [τ2-Bench: Evaluating Conversational Agents in a Dual-Control Environment](https://mlsanity.com/paper/2506.07982/) (2025)
- [API-Bank: A Comprehensive Benchmark for Tool-Augmented LLMs](https://mlsanity.com/paper/2304.08244/) (2023)
- [FinQA: A Dataset of Numerical Reasoning over Financial Data](https://mlsanity.com/paper/2109.00122/) (2021)
- [TAT-QA: A Question Answering Benchmark on a Hybrid of Tabular and Textual Content in Finance](https://mlsanity.com/paper/2105.07624/) (2021)
- [FinanceBench: A New Benchmark for Financial Question Answering](https://mlsanity.com/paper/2311.11944/) (2023)
- [SafeArena: Evaluating the Safety of Autonomous Web Agents](https://mlsanity.com/paper/2503.04957/) (2025)
- [Finance Agent Benchmark: Benchmarking LLMs on Real-world Financial Research Tasks](https://mlsanity.com/paper/2508.00828/) (2025)
- [Rating Roulette: Self-Inconsistency in LLM-As-A-Judge Frameworks](https://mlsanity.com/paper/2510.27106/) (2025)
- [DeepResearch Arena: The First Exam of LLMs' Research Abilities via Seminar-Grounded Tasks](https://mlsanity.com/paper/2509.01396/) (2025)
- [SafeToolBench: Pioneering a Prospective Benchmark to Evaluating Tool Utilization Safety in LLMs](https://mlsanity.com/paper/2509.07315/) (2025)
- [Crowd Comparative Reasoning: Unlocking Comprehensive Evaluations for LLM-as-a-Judge](https://mlsanity.com/paper/2502.12501/) (2025)
- [YESciEval: Robust LLM-as-a-Judge for Scientific Question Answering](https://mlsanity.com/paper/2505.14279/) (2025)
- [Beyond Static Tools: Test-Time Tool Evolution for Scientific Reasoning](https://mlsanity.com/paper/2601.07641/) (2026)
- [From What to Why: A Multi-Agent System for Evidence-based Chemical Reaction Condition Reasoning](https://mlsanity.com/paper/2509.23768/) (2025)
- [FLAME: Financial Large-Language Model Assessment and Metrics Evaluation](https://mlsanity.com/paper/2501.06211/) (2025)
- [SCP: Accelerating Discovery with a Global Web of Autonomous Scientific Agents](https://mlsanity.com/paper/2512.24189/) (2025)

## Built on by (most cited first, 10 of 10)

- [Agentic Trading: When LLM Agents Meet Financial Markets](https://mlsanity.com/paper/2605.19337/) (2026)
- [FinTradeBench: A Financial Reasoning Benchmark for LLMs](https://mlsanity.com/paper/2603.19225/) (2026)
- [FinTrace: Holistic Trajectory-Level Evaluation of LLM Tool Calling for Long-Horizon Financial Tasks](https://mlsanity.com/paper/2604.10015/) (2026)
- [The Alpha Illusion: Reported Alpha from LLM Trading Agents Should Not Be Treated as Deployment Evidence](https://mlsanity.com/paper/2605.16895/) (2026)
- [DSA: Evidence-Aware LLM-Agent Orchestration for Multi-Market Stock Research](https://mlsanity.com/paper/2608.26990/) (2026)
- [Mint-Agent: Introducing Finance-Native Agentic Foundation Models](https://mlsanity.com/paper/2608.16386/) (2026)
- [IndicBankBench: Evaluating Safety and Reliability of Language Model Assistants in Indian Retail Banking](https://mlsanity.com/paper/2609.29167/) (2026)
- [Knowledge or Calculator? Decomposing the Skill Premium in Verifiable Financial Agent Workflows](https://mlsanity.com/paper/2610.03564/) (2026)
- [BazaarBench: Delegation Safety in Decentralized C2C Marketplaces Run by LLM Agents](https://mlsanity.com/paper/2610.06748/) (2026)
- [AlphaPilot: Multi-Turn Tool-Using LLM Agents for Adaptive Alpha Mining with Reinforcement Learning](https://mlsanity.com/paper/2610.15488/) (2026)

## Abstract (arXiv)

The integration of Large Language Models (LLMs) into the financial domain is driving a paradigm shift from passive information retrieval to dynamic, agentic interaction. While general-purpose tool learning has witnessed a surge in benchmarks, the financial sector, characterized by high stakes, strict compliance, and rapid data volatility, remains critically underserved. Existing financial evaluations predominantly focus on static textual analysis or document-based QA, ignoring the complex reality of tool execution. Conversely, general tool benchmarks lack the domain-specific rigor required for finance, often relying on toy environments or a negligible number of financial APIs. To bridge this gap, we introduce FinToolBench, the first real-world, runnable benchmark dedicated to evaluating financial tool learning agents. Unlike prior works limited to a handful of mock tools, FinToolBench establishes a realistic ecosystem coupling 760 executable financial tools with 295 rigorous, tool-required queries. We propose a novel evaluation framework that goes beyond binary execution success, assessing agents on finance-critical dimensions: timeliness, intent type, and regulatory domain alignment. Furthermore, we present FATR, a finance-aware tool retrieval and reasoning baseline that enhances stability and compliance. By providing the first testbed for auditable, agentic financial execution, FinToolBench sets a new standard for trustworthy AI in finance. The tool manifest, execution environment, and evaluation code will be open-sourced to facilitate future research.