# XFinBench: Benchmarking LLMs in Complex Financial Problem Solving and Reasoning

arXiv: https://arxiv.org/abs/2508.15861 · HTML: https://mlsanity.com/paper/2508.15861/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- Placed in Reasoning for agents, planning and code, one of 17 directions in LLM reasoning.
- Builds on 10 of its 45 references that are in the map.
- First posted on arXiv 20 Aug 2025.

## Placed in

- LLM reasoning / [Reasoning for agents, planning and code](https://mlsanity.com/lr/reasoning-for-agents-planning-and-code/)

## Builds on (most cited first, 10 of 10)

- [MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmark](https://mlsanity.com/paper/2406.01574/) (2024)
- [Mixtral of Experts](https://mlsanity.com/paper/2401.04088/) (2024)
- [DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model](https://mlsanity.com/paper/2405.04434/) (2024)
- [Chameleon: Mixed-Modal Early-Fusion Foundation Models](https://mlsanity.com/paper/2405.09818/) (2024)
- [Gemini: A Family of Highly Capable Multimodal Models](https://mlsanity.com/paper/2312.11805/) (2023)
- [Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMs](https://mlsanity.com/paper/2406.16860/) (2024)
- [CharXiv: Charting Gaps in Realistic Chart Understanding in Multimodal LLMs](https://mlsanity.com/paper/2406.18521/) (2024)
- [Vision language models are blind: Failing to translate detailed visual features into words](https://mlsanity.com/paper/2407.06581/) (2024)
- [FinEval: A Chinese Financial Domain Knowledge Evaluation Benchmark for Large Language Models](https://mlsanity.com/paper/2308.09975/) (2023)
- [BizBench: A Quantitative Reasoning Benchmark for Business and Finance](https://mlsanity.com/paper/2311.06602/) (2023)

## Built on by (most cited first, 12 of 12)

- [Finch: Benchmarking Finance & Accounting across Spreadsheet-Centric Enterprise Workflows](https://mlsanity.com/paper/2512.13168/) (2025)
- [Herculean: An Agentic Benchmark for Financial Intelligence](https://mlsanity.com/paper/2605.14355/) (2026)
- [MBABench: Evaluating LLM Agents on End-to-End Spreadsheet Tasks in Finance](https://mlsanity.com/paper/2605.22664/) (2026)
- [FrontierFinance: A Long-Horizon Computer-Use Benchmark of Real-World Financial Tasks](https://mlsanity.com/paper/2604.05912/) (2026)
- [Mind2Report: Expert-Level Commercial Report Synthesis via Cognitive Deep Research Agent](https://mlsanity.com/paper/2601.04879/) (2026)
- [TxSum: User-Centered Ethereum Transaction Understanding with Micro-Level Semantic Grounding](https://mlsanity.com/paper/2512.06933/) (2025)
- [From Natural Language to Executable Option Strategies via Large Language Models](https://mlsanity.com/paper/2603.16434/) (2026)
- [IndustryCode: A Benchmark for Industry Code Generation](https://mlsanity.com/paper/2604.02729/) (2026)
- [UniFinEval: Towards Unified Evaluation of Financial Multimodal Models across Text, Images and Videos](https://mlsanity.com/paper/2601.22162/) (2026)
- [Evaluating Investment Logic in Large Language Models: A Real-World Benchmark Towards Personalzied Financial Agents](https://mlsanity.com/paper/2608.06108/) (2026)
- [CLAIR-Fin: An Adversarial Multi-Agent Framework for Claim-Level Verification and Adaptive Debate in Cross-Modal Financial QA](https://mlsanity.com/paper/2608.13706/) (2026)
- [FinExam-10K: When Retrieval Helps Financial Reasoning?](https://mlsanity.com/paper/2608.28155/) (2026)

## Abstract (arXiv)

Solving financial problems demands complex reasoning, multimodal data processing, and a broad technical understanding, presenting unique challenges for current large language models (LLMs). We introduce XFinBench, a novel benchmark with 4,235 examples designed to evaluate LLM's ability in solving complex, knowledge-intensive financial problems across diverse graduate-level finance topics with multi-modal context. We identify five core capabilities of LLMs using XFinBench, i.e, terminology understanding, temporal reasoning, future forecasting, scenario planning, and numerical modelling. Upon XFinBench, we conduct extensive experiments on 18 leading models. The result shows that o1 is the best-performing text-only model with an overall accuracy of 67.3%, but still lags significantly behind human experts with 12.5%, especially in temporal reasoning and scenario planning capabilities. We further construct a knowledge bank with 3,032 finance terms for knowledge augmentation analysis, and find that relevant knowledge to the question only brings consistent accuracy improvements to small open-source model. Additionally, our error analysis reveals that rounding errors during calculation and blindness to position and intersection of curves in the image are two primary issues leading to model's poor performance in calculating and visual-context questions, respectively. Code and dataset are accessible via GitHub: https://github.com/Zhihan72/XFinBench.