# BizFinBench: A Business-Driven Real-World Financial Benchmark for Evaluating LLMs

arXiv: https://arxiv.org/abs/2505.19457 · HTML: https://mlsanity.com/paper/2505.19457/

## Key facts

- Cited by 22 papers in the map (data through 14 Oct 2026).
- Builds on 18 of its 29 references that are in the map.
- First posted on arXiv 26 May 2025.

## Builds on (most cited first, 18 of 18)

- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Qwen Technical Report](https://mlsanity.com/paper/2309.16609/) (2023)
- [A Survey on LLM-as-a-Judge](https://mlsanity.com/paper/2411.15594/) (2024)
- [FinBERT: Financial Sentiment Analysis with Pre-trained Language Models](https://mlsanity.com/paper/1908.10063/) (2019)
- [PIXIU: A Large Language Model, Instruction Data and Evaluation Benchmark for Finance](https://mlsanity.com/paper/2306.05443/) (2023)
- [Leave No Document Behind: Benchmarking Long-Context LLMs with Extended Multi-Doc QA](https://mlsanity.com/paper/2406.17419/) (2024)
- [WHEN FLUE MEETS FLANG: Benchmarks and Large Pre-trained Language Model for Financial Domain](https://mlsanity.com/paper/2211.00083/) (2022)
- [Revolutionizing Finance with LLMs: An Overview of Applications and Insights](https://mlsanity.com/paper/2401.11641/) (2024)
- [InvestLM: A Large Language Model for Investment using Financial Domain Instruction Tuning](https://mlsanity.com/paper/2309.13064/) (2023)
- [A Survey on Large Language Models for Critical Societal Domains: Finance, Healthcare, and Law](https://mlsanity.com/paper/2405.01769/) (2024)
- [DISC-FinLLM: A Chinese Financial Large Language Model based on Multiple Experts Fine-tuning](https://mlsanity.com/paper/2310.15205/) (2023)
- [FinEval: A Chinese Financial Domain Knowledge Evaluation Benchmark for Large Language Models](https://mlsanity.com/paper/2308.09975/) (2023)
- [DianJin-R1: Evaluating and Enhancing Financial Reasoning in Large Language Models](https://mlsanity.com/paper/2504.15716/) (2025)
- [RevisEval: Improving LLM-as-a-Judge via Response-Adapted References](https://mlsanity.com/paper/2410.05193/) (2024)
- [Nexus: An Omni-Perceptive And -Interactive Model for Language, Audio, And Vision](https://mlsanity.com/paper/2503.01879/) (2025)
- [Baichuan4-Finance Technical Report](https://mlsanity.com/paper/2412.15270/) (2024)
- [CGCE: A Chinese Generative Chat Evaluation Benchmark for General and Financial Domains](https://mlsanity.com/paper/2305.14471/) (2023)

## Built on by (most cited first, 22 of 22)

- [StockBench: Can LLM Agents Trade Stocks Profitably In Real-world Markets?](https://mlsanity.com/paper/2510.02209/) (2025)
- [FinSearchComp: Towards a Realistic, Expert-Level Evaluation of Financial Search and Reasoning](https://mlsanity.com/paper/2509.13160/) (2025)
- [StockSim: A Dual-Mode Order-Level Simulator for Evaluating Multi-Agent LLMs in Financial Markets](https://mlsanity.com/paper/2507.09255/) (2025)
- [CryptoBench: A Dynamic Benchmark for Expert-Level Evaluation of LLM Agents in Cryptocurrency](https://mlsanity.com/paper/2512.00417/) (2025)
- [FinTradeBench: A Financial Reasoning Benchmark for LLMs](https://mlsanity.com/paper/2603.19225/) (2026)
- [FinResearchBench II: A Deep Research Benchmark with Consensus-Derived Gold Rubrics for Distinguishing Financial Report Quality](https://mlsanity.com/paper/2607.12252/) (2026)
- [MBABench: Evaluating LLM Agents on End-to-End Spreadsheet Tasks in Finance](https://mlsanity.com/paper/2605.22664/) (2026)
- [PISA: A Pragmatic Psych-Inspired Unified Memory System for Enhanced AI Agency](https://mlsanity.com/paper/2510.15966/) (2025)
- [FinSafetyBench: Evaluating LLM Safety in Real-World Financial Scenarios](https://mlsanity.com/paper/2605.00706/) (2026)
- [EcoGym: Evaluating LLMs for Long-Horizon Plan-and-Execute in Interactive Economies](https://mlsanity.com/paper/2602.09514/) (2026)
- [RETuning: Upgrading Inference-Time Scaling for Stock Movement Prediction with Large Language Models](https://mlsanity.com/paper/2510.21604/) (2025)
- [Market-Bench: Evaluating Large Language Models on Introductory Quantitative Trading and Market Dynamics](https://mlsanity.com/paper/2512.12264/) (2025)
- [Mitigating Hallucination in Financial Retrieval-Augmented Generation via Fine-Grained Knowledge Verification](https://mlsanity.com/paper/2602.05723/) (2026)
- [Financial Instruction Following Evaluation (FIFE)](https://mlsanity.com/paper/2512.08965/) (2025)
- [Is GPT-OSS All You Need? Benchmarking Large Language Models for Financial Intelligence and the Surprising Efficiency Paradox](https://mlsanity.com/paper/2512.14717/) (2025)
- [PyFi: Toward Pyramid-like Financial Image Understanding for VLMs via Adversarial Agents](https://mlsanity.com/paper/2512.14735/) (2025)
- [LATTICE: Evaluating Decision Support Utility of Crypto Agents](https://mlsanity.com/paper/2604.26235/) (2026)
- [FinBoardBench: Benchmarking Dynamic Wealth Management and Strategic Financial Reasoning of LLMs via Board Game Simulations](https://mlsanity.com/paper/2605.27896/) (2026)
- [Absorbing Complexity: An Interaction-Native Knowledge Harness for Financial LLM Agents](https://mlsanity.com/paper/2606.01886/) (2026)
- [Are the Financial Reasoning from LLMs Credible? A Real World Test over Long-Horizon Statements](https://mlsanity.com/paper/2607.28661/) (2026)
- [Mint-Agent: Introducing Finance-Native Agentic Foundation Models](https://mlsanity.com/paper/2608.16386/) (2026)
- [FinFIRST: Benchmarking Search Agents for Financial Information Retrieval, Sourcing and Traceability](https://mlsanity.com/paper/2609.25192/) (2026)

## Abstract (arXiv)

Large language models excel in general tasks, yet assessing their reliability in logic-heavy, precision-critical domains like finance, law, and healthcare remains challenging. To address this, we introduce BizFinBench, the first benchmark specifically designed to evaluate LLMs in real-world financial applications. BizFinBench consists of 6,781 well-annotated queries in Chinese, spanning five dimensions: numerical calculation, reasoning, information extraction, prediction recognition, and knowledge-based question answering, grouped into nine fine-grained categories. The benchmark includes both objective and subjective metrics. We also introduce IteraJudge, a novel LLM evaluation method that reduces bias when LLMs serve as evaluators in objective metrics. We benchmark 25 models, including both proprietary and open-source systems. Extensive experiments show that no model dominates across all tasks. Our evaluation reveals distinct capability patterns: (1) In Numerical Calculation, Claude-3.5-Sonnet (63.18) and DeepSeek-R1 (64.04) lead, while smaller models like Qwen2.5-VL-3B (15.92) lag significantly; (2) In Reasoning, proprietary models dominate (ChatGPT-o3: 83.58, Gemini-2.0-Flash: 81.15), with open-source models trailing by up to 19.49 points; (3) In Information Extraction, the performance spread is the largest, with DeepSeek-R1 scoring 71.46, while Qwen3-1.7B scores 11.23; (4) In Prediction Recognition, performance variance is minimal, with top models scoring between 39.16 and 50.00. We find that while current LLMs handle routine finance queries competently, they struggle with complex scenarios requiring cross-concept reasoning. BizFinBench offers a rigorous, business-aligned benchmark for future research. The code and dataset are available at https://github.com/HiThink-Research/BizFinBench.