# FinEval-KR: A Financial Domain Evaluation Framework for Large Language Models' Knowledge and Reasoning

arXiv: https://arxiv.org/abs/2506.21591 · HTML: https://mlsanity.com/paper/2506.21591/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Placed in Reasoning for agents, planning and code, one of 17 directions in LLM reasoning.
- Builds on 13 of its 28 references that are in the map.
- First posted on arXiv 18 Jun 2025.

## Placed in

- LLM reasoning / [Reasoning for agents, planning and code](https://mlsanity.com/lr/reasoning-for-agents-planning-and-code/)

## Builds on (most cited first, 13 of 13)

- [Judging the Judges: Evaluating Alignment and Vulnerabilities in LLMs-as-Judges](https://mlsanity.com/paper/2406.12624/) (2024)
- [A Survey of Large Language Models for Financial Applications: Progress, Prospects and Challenges](https://mlsanity.com/paper/2406.11903/) (2024)
- [Fin-R1: A Large Language Model for Financial Reasoning through Reinforcement Learning](https://mlsanity.com/paper/2503.16252/) (2025)
- [FinGPT: Instruction Tuning Benchmark for Open-Source Large Language Models in Financial Datasets](https://mlsanity.com/paper/2310.04793/) (2023)
- [INVESTORBENCH: A Benchmark for Financial Decision-Making Tasks with LLM-based Agent](https://mlsanity.com/paper/2412.18174/) (2024)
- [FinEval: A Chinese Financial Domain Knowledge Evaluation Benchmark for Large Language Models](https://mlsanity.com/paper/2308.09975/) (2023)
- [Toward Generalizable Evaluation in the LLM Era: A Survey Beyond Benchmarks](https://mlsanity.com/paper/2504.18838/) (2025)
- [CFBenchmark: Chinese Financial Assistant Benchmark for Large Language Model](https://mlsanity.com/paper/2311.05812/) (2023)
- [Re-TASK: Revisiting LLM Tasks from Capability, Skill, and Knowledge Perspectives](https://mlsanity.com/paper/2408.06904/) (2024)
- [CFinBench: A Comprehensive Chinese Financial Benchmark for Large Language Models](https://mlsanity.com/paper/2407.02301/) (2024)
- [MARS: Benchmarking the Metaphysical Reasoning Abilities of Language Models with a Multi-task Evaluation Dataset](https://mlsanity.com/paper/2406.02106/) (2024)
- [FinTruthQA: A Benchmark for AI-Driven Financial Disclosure Quality Assessment in Investor -- Firm Interactions](https://mlsanity.com/paper/2406.12009/) (2024)
- [MTFinEval:A Multi-domain Chinese Financial Benchmark with Eurypalynous questions](https://mlsanity.com/paper/2408.10921/) (2024)

## Built on by (most cited first, 2 of 2)

- [Market-Bench: Evaluating Large Language Models on Introductory Quantitative Trading and Market Dynamics](https://mlsanity.com/paper/2512.12264/) (2025)
- [Are the Financial Reasoning from LLMs Credible? A Real World Test over Long-Horizon Statements](https://mlsanity.com/paper/2607.28661/) (2026)

## Abstract (arXiv)

Large Language Models (LLMs) demonstrate significant potential but face challenges in complex financial reasoning tasks requiring both domain knowledge and sophisticated reasoning. Current evaluation benchmarks often fall short by not decoupling these capabilities indicators from single task performance and lack root cause analysis for task failure. To address this, we introduce FinEval-KR, a novel evaluation framework for decoupling and quantifying LLMs' knowledge and reasoning abilities independently, proposing distinct knowledge score and reasoning score metrics. Inspired by cognitive science, we further propose a cognitive score based on Bloom's taxonomy to analyze capabilities in reasoning tasks across different cognitive levels. We also release a new open-source Chinese financial reasoning dataset covering 22 subfields to support reproducible research and further advancements in financial reasoning. Our experimental results reveal that LLM reasoning ability and higher-order cognitive ability are the core factors influencing reasoning accuracy. We also specifically find that even top models still face a bottleneck with knowledge application. Furthermore, our analysis shows that specialized financial LLMs generally lag behind the top general large models across multiple metrics.