# FinEval: A Chinese Financial Domain Knowledge Evaluation Benchmark for Large Language Models

arXiv: https://arxiv.org/abs/2308.09975 · HTML: https://mlsanity.com/paper/2308.09975/

## Key facts

- Cited by 52 papers in the map (data through 14 Oct 2026).
- Builds on 31 of its 54 references that are in the map.
- First posted on arXiv 19 Aug 2023.

## Builds on (most cited first, 30 of 31)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [Evaluating Large Language Models Trained on Code](https://mlsanity.com/paper/2107.03374/) (2021)
- [Measuring Massive Multitask Language Understanding](https://mlsanity.com/paper/2009.03300/) (2020)
- [Measuring Mathematical Problem Solving With the MATH Dataset](https://mlsanity.com/paper/2103.03874/) (2021)
- [Large Language Models are Zero-Shot Reasoners](https://mlsanity.com/paper/2205.11916/) (2022)
- [Scaling Instruction-Finetuned Language Models](https://mlsanity.com/paper/2210.11416/) (2022)
- [Training Compute-Optimal Large Language Models](https://mlsanity.com/paper/2203.15556/) (2022)
- [TruthfulQA: Measuring How Models Mimic Human Falsehoods](https://mlsanity.com/paper/2109.07958/) (2021)
- [Self-Instruct: Aligning Language Models with Self-Generated Instructions](https://mlsanity.com/paper/2212.10560/) (2022)
- [Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models](https://mlsanity.com/paper/2206.04615/) (2022)
- [BLOOM: A 176B-Parameter Open-Access Multilingual Language Model](https://mlsanity.com/paper/2211.05100/) (2022)
- [Scaling Language Models: Methods, Analysis & Insights from Training Gopher](https://mlsanity.com/paper/2112.11446/) (2021)
- [GLM: General Language Model Pretraining with Autoregressive Blank Infilling](https://mlsanity.com/paper/2103.10360/) (2021)
- [BloombergGPT: A Large Language Model for Finance](https://mlsanity.com/paper/2303.17564/) (2023)
- [CodeGen: An Open Large Language Model for Code with Multi-Turn Program Synthesis](https://mlsanity.com/paper/2203.13474/) (2022)
- [GLM-130B: An Open Bilingual Pre-trained Model](https://mlsanity.com/paper/2210.02414/) (2022)
- [AGIEval: A Human-Centric Benchmark for Evaluating Foundation Models](https://mlsanity.com/paper/2304.06364/) (2023)
- [C-Eval: A Multi-Level Multi-Discipline Chinese Evaluation Suite for Foundation Models](https://mlsanity.com/paper/2305.08322/) (2023)
- [FinQA: A Dataset of Numerical Reasoning over Financial Data](https://mlsanity.com/paper/2109.00122/) (2021)
- [Efficient and Effective Text Encoding for Chinese LLaMA and Alpaca](https://mlsanity.com/paper/2304.08177/) (2023)
- [CLUE: A Chinese Language Understanding Evaluation Benchmark](https://mlsanity.com/paper/2004.05986/) (2020)
- [ConvFinQA: Exploring the Chain of Numerical Reasoning in Conversational Finance Question Answering](https://mlsanity.com/paper/2210.03849/) (2022)
- [Evaluating the Performance of Large Language Models on GAOKAO Benchmark](https://mlsanity.com/paper/2305.12474/) (2023)
- [WHEN FLUE MEETS FLANG: Benchmarks and Large Pre-trained Language Model for Financial Domain](https://mlsanity.com/paper/2211.00083/) (2022)
- [CBLUE: A Chinese Biomedical Language Understanding Evaluation Benchmark](https://mlsanity.com/paper/2106.08087/) (2021)
- [XuanYuan 2.0: A Large Chinese Financial Chat Model with Hundreds of Billions Parameters](https://mlsanity.com/paper/2305.12002/) (2023)

## Built on by (most cited first, 30 of 52)

- [LawBench: Benchmarking Legal Knowledge of Large Language Models](https://mlsanity.com/paper/2309.16289/) (2023)
- [A Survey of Large Language Models for Financial Applications: Progress, Prospects and Challenges](https://mlsanity.com/paper/2406.11903/) (2024)
- [Fin-R1: A Large Language Model for Financial Reasoning through Reinforcement Learning](https://mlsanity.com/paper/2503.16252/) (2025)
- [StockBench: Can LLM Agents Trade Stocks Profitably In Real-world Markets?](https://mlsanity.com/paper/2510.02209/) (2025)
- [BizFinBench: A Business-Driven Real-World Financial Benchmark for Evaluating LLMs](https://mlsanity.com/paper/2505.19457/) (2025)
- [Mixing It Up: The Cocktail Effect of Multi-Task Fine-Tuning on LLM Performance -- A Case Study in Finance](https://mlsanity.com/paper/2410.01109/) (2024)
- [LiveTradeBench: Seeking Real-World Alpha with Large Language Models](https://mlsanity.com/paper/2511.03628/) (2025)
- [A Comprehensive Survey of Contamination Detection Methods in Large Language Models](https://mlsanity.com/paper/2404.00699/) (2024)
- [CFinBench: A Comprehensive Chinese Financial Benchmark for Large Language Models](https://mlsanity.com/paper/2407.02301/) (2024)
- [Do We Need Domain-Specific Embedding Models? An Empirical Investigation](https://mlsanity.com/paper/2409.18511/) (2024)
- [XFinBench: Benchmarking LLMs in Complex Financial Problem Solving and Reasoning](https://mlsanity.com/paper/2508.15861/) (2025)
- [CFGPT: Chinese Financial Assistant with Large Language Model](https://mlsanity.com/paper/2309.10654/) (2023)
- [LogEval: A Comprehensive Benchmark Suite for Large Language Models In Log Analysis](https://mlsanity.com/paper/2407.01896/) (2024)
- [Fin-RATE: A Real-world Financial Analytics and Tracking Evaluation Benchmark for LLMs on SEC Filings](https://mlsanity.com/paper/2602.07294/) (2026)
- [Dr.Academy: A Benchmark for Evaluating Questioning Capability in Education for Large Language Models](https://mlsanity.com/paper/2408.10947/) (2024)
- [EDINET-Bench: Evaluating LLMs on Complex Financial Tasks using Japanese Financial Statements](https://mlsanity.com/paper/2506.08762/) (2025)
- [Agentar-Fin-R1: Enhancing Financial Intelligence through Domain Expertise, Training Efficiency, and Advanced Reasoning](https://mlsanity.com/paper/2507.16802/) (2025)
- [AECBench: A Hierarchical Benchmark for Knowledge Evaluation of Large Language Models in the AEC Field](https://mlsanity.com/paper/2509.18776/) (2025)
- [OpsEval: A Comprehensive IT Operations Benchmark Suite for Large Language Models](https://mlsanity.com/paper/2310.07637/) (2023)
- [Enterprise Large Language Model Evaluation Benchmark](https://mlsanity.com/paper/2506.20274/) (2025)
- [FinTeam: A Multi-Agent Collaborative Intelligence System for Comprehensive Financial Scenarios](https://mlsanity.com/paper/2507.10448/) (2025)
- [FIRE: A Comprehensive Benchmark for Financial Intelligence and Reasoning Evaluation](https://mlsanity.com/paper/2602.22273/) (2026)
- [Golden Touchstone: A Comprehensive Bilingual Benchmark for Evaluating Financial Large Language Models](https://mlsanity.com/paper/2411.06272/) (2024)
- [Advancing Financial Engineering with Foundation Models: Progress, Applications, and Challenges](https://mlsanity.com/paper/2507.18577/) (2025)
- [Unlocking Data Value in Finance: A Study on Distillation and Difficulty-Aware Training](https://mlsanity.com/paper/2603.07223/) (2026)
- [Financial Knowledge Large Language Model](https://mlsanity.com/paper/2407.00365/) (2024)
- [LegalEval-Q: A New Benchmark for The Quality Evaluation of LLM-Generated Legal Text](https://mlsanity.com/paper/2505.24826/) (2025)
- [MSCoRe: A Benchmark for Multi-Stage Collaborative Reasoning in LLM Agents](https://mlsanity.com/paper/2509.17628/) (2025)
- [CPSDBench: A Large Language Model Evaluation Benchmark and Baseline for Chinese Public Security Domain](https://mlsanity.com/paper/2402.07234/) (2024)
- [FinSphere, a Real-Time Stock Analysis Agent Powered by Instruction-Tuned LLMs and Domain Tools](https://mlsanity.com/paper/2501.12399/) (2025)

## Abstract (arXiv)

Large language models have demonstrated outstanding performance in various natural language processing tasks, but their security capabilities in the financial domain have not been explored, and their performance on complex tasks like financial agent remains unknown. This paper presents FinEval, a benchmark designed to evaluate LLMs' financial domain knowledge and practical abilities. The dataset contains 8,351 questions categorized into four different key areas: Financial Academic Knowledge, Financial Industry Knowledge, Financial Security Knowledge, and Financial Agent. Financial Academic Knowledge comprises 4,661 multiple-choice questions spanning 34 subjects such as finance and economics. Financial Industry Knowledge contains 1,434 questions covering practical scenarios like investment research. Financial Security Knowledge assesses models through 1,640 questions on topics like application security and cryptography. Financial Agent evaluates tool usage and complex reasoning with 616 questions. FinEval has multiple evaluation settings, including zero-shot, five-shot with chain-of-thought, and assesses model performance using objective and subjective criteria. Our results show that Claude 3.5-Sonnet achieves the highest weighted average score of 72.9 across all financial domain categories under zero-shot setting. Our work provides a comprehensive benchmark closely aligned with Chinese financial domain.