When Tables Go Crazy: Evaluating Multimodal Models on French Financial Documents
On the map: Multimodal LLMs / vision-language models → Chart understanding and anomaly detection
Abstract (arXiv)
Vision-language models (VLMs) perform well on many document understanding tasks, yet their reliability in specialized, non-English domains remains underexplored. This gap is especially critical in finance, where documents mix dense regulatory text, numerical tables, and visual charts, and where extraction errors can have real-world consequences. We introduce Scribe Finance, the first multimodal benchmark for evaluating French financial document understanding. The dataset contains 1,204 expert-validated questions spanning text extraction, table comprehension, chart interpretation, and multi-turn conversational reasoning, drawn from real investment prospectuses, KIDs, and PRIIPs. We evaluate six open-weight VLMs (8B-124B parameters) using an LLM-as-judge protocol. While models achieve strong performance on text and table tasks (85-90% accuracy), they struggle with chart interpretation (34-62%). Most notably, multi-turn dialogue reveals a sharp failure mode: early mistakes propagate across turns, driving accuracy down to roughly 50% regardless of model size. These results show that current VLMs are effective for well-defined extraction tasks but remain brittle in interactive, multi-step financial analysis. Scribe Finance offers a challenging benchmark to measure and drive progress in this high-stakes setting.
Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Chart understanding and anomaly detection, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 14 of its 31 references that are in the map. First posted on arXiv 11 Feb 2026.
Builds on
14 of 31 references.
References · 14
- 2024The Llama 3 Herd of Modelscited by 14,192
- 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
- 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,115
- 2020CrowS-Pairs: A Challenge Dataset for Measuring Social Biases in Masked Language Modelscited by 703
- 2021FinQA: A Dataset of Numerical Reasoning over Financial Datacited by 552
- 2021TAT-QA: A Question Answering Benchmark on a Hybrid of Tabular and Textual Content in Financecited by 402
- 2022ConvFinQA: Exploring the Chain of Numerical Reasoning in Conversational Finance Question Answeringcited by 194
- 2020MKQA: A Linguistically Diverse Benchmark for Multilingual Open Domain Question Answeringcited by 151
- 2026Molmo2: Open Weights and Data for Vision-Language Models with Video Understanding and Groundingcited by 132
- 2020FQuAD: French Question Answering Datasetcited by 58
- 2022PACIFIC: Towards Proactive Conversational Question Answering over Tabular and Textual Data in Financecited by 45
- 2025Beyond the Surface: Measuring Self-Preference in LLM Judgmentscited by 30
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2025FinMMDocR: Benchmarking Financial Multimodal Reasoning with Scenario Awareness, Document Understanding, and Multi-Step Computation
- 2025FinMR: A Knowledge-Intensive Multimodal Benchmark for Advanced Financial Reasoning
- 2025FinCriticalED: A Visual Benchmark for Financial Fact-Level OCR
- 2026Confidence Estimation for Financial Vision-Language Models in Chart and Document Understanding
- 2026Are Large Language Models Reliable Reviewers? A Benchmark for Error Detection in Financial Documents
- 2026FinMTM: A Multi-Turn Multimodal Benchmark for Financial Reasoning and Agent Evaluation
- 2026Benchmarking Large Vision-Language Models on CFMME: A Comprehensive Chinese Financial Multimodal Evaluation Dataset
- 2025CFBenchmark-MM: Chinese Financial Assistant Benchmark for Multimodal Large Language Model