# FinAuditing: A Financial Taxonomy-Structured Multi-Document Benchmark for Evaluating LLMs

arXiv: https://arxiv.org/abs/2510.08886 · HTML: https://mlsanity.com/paper/2510.08886/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- Builds on 13 of its 45 references that are in the map.
- First posted on arXiv 10 Oct 2025.

## Builds on (most cited first, 13 of 13)

- [GPT-4o System Card](https://mlsanity.com/paper/2410.21276/) (2024)
- [A Survey on LLM-as-a-Judge](https://mlsanity.com/paper/2411.15594/) (2024)
- [Neural Collaborative Filtering](https://mlsanity.com/paper/1708.05031/) (2017)
- [FinQA: A Dataset of Numerical Reasoning over Financial Data](https://mlsanity.com/paper/2109.00122/) (2021)
- [LLMs-as-Judges: A Comprehensive Survey on LLM-based Evaluation Methods](https://mlsanity.com/paper/2412.05579/) (2024)
- [TAT-QA: A Question Answering Benchmark on a Hybrid of Tabular and Textual Content in Finance](https://mlsanity.com/paper/2105.07624/) (2021)
- [PIXIU: A Large Language Model, Instruction Data and Evaluation Benchmark for Finance](https://mlsanity.com/paper/2306.05443/) (2023)
- [ConvFinQA: Exploring the Chain of Numerical Reasoning in Conversational Finance Question Answering](https://mlsanity.com/paper/2210.03849/) (2022)
- [MultiHiertt: Numerical Reasoning over Multi Hierarchical Tabular and Textual Data](https://mlsanity.com/paper/2206.01347/) (2022)
- [Impact of News on the Commodity Market: Dataset and Results](https://mlsanity.com/paper/2009.04202/) (2020)
- [FiNER: Financial Numeric Entity Recognition for XBRL Tagging](https://mlsanity.com/paper/2203.06482/) (2022)
- [DocMath-Eval: Evaluating Math Reasoning Capabilities of LLMs in Understanding Long and Specialized Documents](https://mlsanity.com/paper/2311.09805/) (2023)
- [Financial Numeric Extreme Labelling: A Dataset and Benchmarking for XBRL Tagging](https://mlsanity.com/paper/2306.03723/) (2023)

## Built on by (most cited first, 14 of 14)

- [Finch: Benchmarking Finance & Accounting across Spreadsheet-Centric Enterprise Workflows](https://mlsanity.com/paper/2512.13168/) (2025)
- [The Illusion of Specialization: Unveiling the Domain-Invariant "Standing Committee" in Mixture-of-Experts Models](https://mlsanity.com/paper/2601.03425/) (2026)
- [Can LLM Agents Be CFOs? Benchmarking Long-Horizon Resource Allocation in an Uncertain Enterprise Environment](https://mlsanity.com/paper/2603.23638/) (2026)
- [Herculean: An Agentic Benchmark for Financial Intelligence](https://mlsanity.com/paper/2605.14355/) (2026)
- [FinBalance: A Multi-Document Accounting Reconciliation Benchmark](https://mlsanity.com/paper/2606.15949/) (2026)
- [FinCriticalED: A Visual Benchmark for Financial Fact-Level OCR](https://mlsanity.com/paper/2511.14998/) (2025)
- [AUDITFLOW: Executable Symbolic Environments for Structured Financial Reporting Verification](https://mlsanity.com/paper/2606.03031/) (2026)
- [Evaluating Financial Intelligence in Large Language Models: Benchmarking SuperInvesting AI with LLM Engines](https://mlsanity.com/paper/2603.08704/) (2026)
- [FinReasoning: A Hierarchical Benchmark for Reliable Financial Research Reporting](https://mlsanity.com/paper/2603.19254/) (2026)
- [On Finding Inconsistencies in Documents](https://mlsanity.com/paper/2512.18601/) (2025)
- [FinVerBench: Benchmark Validity and Calibration in Large Language Model Financial Statement Verification](https://mlsanity.com/paper/2605.29586/) (2026)
- [FinRCA-Bench: Benchmarking Evidence Retrieval and Reasoning for Financial AI Systems](https://mlsanity.com/paper/2608.18534/) (2026)
- [VERA-8B: Evidence-Grounded Audit Risk Reasoning from SEC Filings](https://mlsanity.com/paper/2608.28402/) (2026)
- [FinAutoRubric: Expert-Guided Automatic Rubric Generation for Evaluating Financial Research Agents](https://mlsanity.com/paper/2609.35744/) (2026)

## Abstract (arXiv)

Going beyond simple text processing, financial auditing requires detecting semantic, structural, and numerical inconsistencies across large-scale disclosures. As financial reports are filed in XBRL, a structured XML format governed by accounting standards, auditing becomes a structured information extraction and reasoning problem involving concept alignment, taxonomy-defined relations, and cross-document consistency. Although large language models (LLMs) show promise on isolated financial tasks, their capability in professional-grade auditing remains unclear. We introduce FinAuditing, a taxonomy-aligned, structure-aware benchmark built from real XBRL filings. It contains 1,102 annotated instances averaging over 33k tokens and defines three tasks: Financial Semantic Matching (FinSM), Financial Relationship Extraction (FinRE), and Financial Mathematical Reasoning (FinMR). Evaluations of 13 state-of-the-art LLMs reveal substantial gaps in concept retrieval, taxonomy-aware relation modeling, and consistent cross-document reasoning. These findings highlight the need for realistic, structure-aware benchmarks. We release the evaluation code at https://github.com/The-FinAI/FinAuditing and the dataset at https://huggingface.co/collections/TheFinAI/finauditing. The task currently serves as the official benchmark of an ongoing public evaluation contest at https://open-finance-lab.github.io/SecureFinAI_Contest_2026/.