# BizFinBench.v2: Towards Reliable LLMs in Finance via Real-User Data and Offline/Online Bilingual Evaluation

arXiv: https://arxiv.org/abs/2601.06401 · HTML: https://mlsanity.com/paper/2601.06401/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 28 of its 44 references that are in the map.
- First posted on arXiv 10 Jan 2026.

## Builds on (most cited first, 28 of 28)

- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [ChatGLM: A Family of Large Language Models from GLM-130B to GLM-4 All Tools](https://mlsanity.com/paper/2406.12793/) (2024)
- [FinQA: A Dataset of Numerical Reasoning over Financial Data](https://mlsanity.com/paper/2109.00122/) (2021)
- [PIXIU: A Large Language Model, Instruction Data and Evaluation Benchmark for Finance](https://mlsanity.com/paper/2306.05443/) (2023)
- [ConvFinQA: Exploring the Chain of Numerical Reasoning in Conversational Finance Question Answering](https://mlsanity.com/paper/2210.03849/) (2022)
- [FinMem: A Performance-Enhanced LLM Trading Agent with Layered Memory and Character Design](https://mlsanity.com/paper/2311.13743/) (2023)
- [FinBen: A Holistic Financial Benchmark for Large Language Models](https://mlsanity.com/paper/2402.12659/) (2024)
- [FinCon: A Synthesized LLM Multi-Agent System with Conceptual Verbal Reinforcement for Enhanced Financial Decision Making](https://mlsanity.com/paper/2407.06567/) (2024)
- [Fin-R1: A Large Language Model for Financial Reasoning through Reinforcement Learning](https://mlsanity.com/paper/2503.16252/) (2025)
- [InvestLM: A Large Language Model for Investment using Financial Domain Instruction Tuning](https://mlsanity.com/paper/2309.13064/) (2023)
- [A Survey on Large Language Models for Critical Societal Domains: Finance, Healthcare, and Law](https://mlsanity.com/paper/2405.01769/) (2024)
- [DISC-FinLLM: A Chinese Financial Large Language Model based on Multiple Experts Fine-tuning](https://mlsanity.com/paper/2310.15205/) (2023)
- [Instruct-FinGPT: Financial Sentiment Analysis by Instruction Tuning of General-Purpose Large Language Models](https://mlsanity.com/paper/2306.12659/) (2023)
- [Alpha-GPT: Human-AI Interactive Alpha Mining for Quantitative Investment](https://mlsanity.com/paper/2308.00016/) (2023)
- [AlphaFin: Benchmarking Financial Analysis with Retrieval-Augmented Stock-Chain Framework](https://mlsanity.com/paper/2403.12582/) (2024)
- [FinanceMath: Knowledge-Intensive Math Reasoning in Finance Domains](https://mlsanity.com/paper/2311.09797/) (2023)
- [Fino1: On the Transferability of Reasoning-Enhanced LLMs and Reinforcement Learning to Finance](https://mlsanity.com/paper/2502.08127/) (2025)
- [QuantAgent: Seeking Holy Grail in Trading by Self-Improving Large Language Model](https://mlsanity.com/paper/2402.03755/) (2024)
- [FinanceReasoning: Benchmarking Financial Numerical Reasoning More Credible, Comprehensive and Challenging](https://mlsanity.com/paper/2506.05828/) (2025)
- [Benchmarking Large Language Models on CFLUE -- A Chinese Financial Language Understanding Evaluation Dataset](https://mlsanity.com/paper/2405.10542/) (2024)
- [DianJin-R1: Evaluating and Enhancing Financial Reasoning in Large Language Models](https://mlsanity.com/paper/2504.15716/) (2025)
- [Nexus: An Omni-Perceptive And -Interactive Model for Language, Audio, And Vision](https://mlsanity.com/paper/2503.01879/) (2025)
- [FinReport: Explainable Stock Earnings Forecasting via News Factor Analyzing Model](https://mlsanity.com/paper/2403.02647/) (2024)
- [StockGPT: A GenAI Model for Stock Prediction and Trading](https://mlsanity.com/paper/2404.05101/) (2024)
- [FinMaster: A Holistic Benchmark for Mastering Full-Pipeline Financial Workflows with LLMs](https://mlsanity.com/paper/2505.13533/) (2025)
- [CGCE: A Chinese Generative Chat Evaluation Benchmark for General and Financial Domains](https://mlsanity.com/paper/2305.14471/) (2023)
- [FinVault: Benchmarking Financial Agent Safety in Execution-Grounded Environments](https://mlsanity.com/paper/2601.07853/) (2026)
- [VisFinEval: A Scenario-Driven Chinese Multimodal Benchmark for Holistic Financial Understanding](https://mlsanity.com/paper/2508.09641/) (2025)

## Abstract (arXiv)

Large language models are becoming increasingly significant in financial applications. Nevertheless, prevailing benchmarks are largely dependent on simulated or generic data, which leads to a significant gap between reported performance and actual efficacy in real-world scenarios. To tackle this challenge, we present BizFinBench.v2, the first integrated offline and online benchmark built upon authentic user query-response data from both Chinese and U.S. equity markets. It comprises 28,860 questions across eight offline and two online tasks. Experimental results show that GPT-5 achieves a mere 61.5% accuracy, still failing to meet the practical business requirement (84.8%). Among the evaluated commercial models, DeepSeek-R1 exhibits superior investment efficacy. Error analysis grounded in real financial practice reveals persistent limitations in existing models. By overcoming the constraints of prior benchmarks, BizFinBench.v2 provides a substantiated foundation for advancing LLM deployment in the financial sector. Our data and code are available at https://github.com/HiThink-Research/BizFinBench.v2.