mlsanityData to 7 Oct 2026

FinMMR: Make Financial Numerical Reasoning More Multimodal, Comprehensive, and Challenging

arXiv 2508.04625 ↗First posted 6 Aug 2025cs.CV cs.CEcited by 6

On the map: LLM reasoning → Reasoning for agents, planning and code

also in 1 more
Abstract (arXiv)

We present FinMMR, a novel bilingual multimodal benchmark tailored to evaluate the reasoning capabilities of multimodal large language models (MLLMs) in financial numerical reasoning tasks. Compared to existing benchmarks, our work introduces three significant advancements. (1) Multimodality: We meticulously transform existing financial reasoning benchmarks, and construct novel questions from the latest Chinese financial research reports. FinMMR comprises 4.3K questions and 8.7K images spanning 14 categories, including tables, bar charts, and ownership structure charts. (2) Comprehensiveness: FinMMR encompasses 14 financial subdomains, including corporate finance, banking, and industry analysis, significantly exceeding existing benchmarks in financial domain knowledge breadth. (3) Challenge: Models are required to perform multi-step precise numerical reasoning by integrating financial knowledge with the understanding of complex financial images and text. The best-performing MLLM achieves only 53.0% accuracy on Hard problems. We believe that FinMMR will drive advancements in enhancing the reasoning capabilities of MLLMs in real-world scenarios.

Key facts. Cited by 6 papers in the map (data through 7 Oct 2026). Placed in Reasoning for agents, planning and code, one of 17 directions in LLM reasoning. Builds on 47 of its 63 references that are in the map. First posted on arXiv 6 Aug 2025.

Builds on

47 of 63 references.

Closest ancestors · 8

Same direction, newest first

  1. 2025FinanceReasoning: Benchmarking Financial Numerical Reasoning More Credible, Comprehensive and Challenging6 Jun
  2. 2025FinMME: Benchmark Dataset for Financial Multi-Modal Reasoning Evaluation30 May
  3. 2024MME-Finance: A Multimodal Finance Benchmark for Expert-level Understanding and Reasoning5 Nov
  4. 2024FAMMA: A Benchmark for Financial Domain Multilingual Multimodal Question Answering6 Oct
  5. 2024LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code12 Mar
  6. 2023FinanceMath: Knowledge-Intensive Math Reasoning in Finance Domains16 Nov
  7. 2023BizBench: A Quantitative Reasoning Benchmark for Business and Finance11 Nov
  8. 2022ConvFinQA: Exploring the Chain of Numerical Reasoning in Conversational Finance Question Answering7 Oct

Other references · 39

  1. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  2. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  3. 2020Scaling Laws for Neural Language Modelscited by 6,411
  4. 2025Qwen2.5-VL Technical Reportcited by 5,418
  5. 2023Let's Verify Step by Stepcited by 3,598
  6. 2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,579
  7. 2023Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyondcited by 2,179
  8. 2023MMBench: Is Your Multi-modal Model an All-around Player?cited by 2,105
  9. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,044
  10. 2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Modelscited by 1,624
  11. 2024Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scalingcited by 1,573
  12. 2023MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilitiescited by 1,221
27 more
  1. 2021Unsupervised Dense Information Retrieval with Contrastive Learningcited by 1,203
  2. 2025Kimi k1.5: Scaling Reinforcement Learning with LLMscited by 1,193
  3. 2023GAIA: a benchmark for General AI Assistantscited by 1,084
  4. 2022Program of Thoughts Prompting: Disentangling Computation from Reasoning for Numerical Reasoning Taskscited by 1,057
  5. 2024Are We on the Right Way for Evaluating Large Vision-Language Models?cited by 893
  6. 2024MMMU-Pro: A More Robust Multi-discipline Multimodal Understanding Benchmarkcited by 494
  7. 2025Search-o1: Agentic Search-Enhanced Large Reasoning Modelscited by 423
  8. 2021TAT-QA: A Question Answering Benchmark on a Hybrid of Tabular and Textual Content in Financecited by 402
  9. 2023Detecting Pretraining Data from Large Language Modelscited by 375
  10. 2023Check Your Facts and Try Again: Improving Large Language Models with External Knowledge and Automated Feedbackcited by 351
  11. 2023NLP Evaluation in trouble: On the Need to Measure LLM Data Contamination for each Benchmarkcited by 339
  12. 2025Towards Large Reasoning Models: A Survey of Reinforced Reasoning with Large Language Modelscited by 291
  13. 2023SciBench: Evaluating College-Level Scientific Problem-Solving Abilities of Large Language Modelscited by 246
  14. 2023OpenAGI: When LLM Meets Domain Expertscited by 196
  15. 2023MMC: Advancing Multimodal Chart Understanding with Large-scale Instruction Tuningcited by 149
  16. 2023Investigating Data Contamination in Modern Benchmarks for Large Language Modelscited by 136
  17. 2025Can MLLMs Reason in Multimodality? EMMA: An Enhanced MultiModal ReAsoning Benchmarkcited by 132
  18. 2025MME-CoT: Benchmarking Chain-of-Thought in Large Multimodal Models for Reasoning Quality, Robustness, and Efficiencycited by 123
  19. 2024GMAI-MMBench: A Comprehensive Multimodal Evaluation Benchmark Towards General Medical AIcited by 96
  20. 2023Levels of AGI for Operationalizing Progress on the Path to AGIcited by 94
  21. 2023DocMath-Eval: Evaluating Math Reasoning Capabilities of LLMs in Understanding Long and Specialized Documentscited by 45
  22. 2024CHAMP: A Competition-level Dataset for Fine-Grained Analyses of LLMs' Mathematical Reasoning Capabilitiescited by 43
  23. 2023Beyond Factuality: A Comprehensive Evaluation of Large Language Models as Knowledge Generatorscited by 42
  24. 2024MIBench: Evaluating Multimodal Large Language Models over Multiple Imagescited by 36
  25. 2024AMOR: A Recipe for Building Adaptable Modular Knowledge Agents Through Process Feedbackcited by 25
  26. 2024Losing Visual Needles in Image Haystacks: Vision Language Models are Easily Distracted in Short and Long Contextscited by 14
  27. 2024Plan*RAG: Efficient Test-Time Planning for Retrieval Augmented Generationcited by 11

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 1; 2026: 5.

6 papers.

  1. 2026QuantaAlpha: An Evolutionary Framework for LLM-Driven Alpha Miningcited by 12
  2. 2026Who Deserves the Reward? SHARP: Shapley Credit-based Optimization for Multi-Agent Systemcited by 7
  3. 2025PyFi: Toward Pyramid-like Financial Image Understanding for VLMs via Adversarial Agentscited by 0
  4. 2026UniFinEval: Towards Unified Evaluation of Financial Multimodal Models across Text, Images and Videoscited by 0
  5. 2026Beyond Single-Modal Analytics: A Framework for Integrating Heterogeneous LLM-Based Query Systems for Multi-Modal Datacited by 0
  6. 2026ProjLens: Unveiling the Role of Projectors in Multimodal Model Safetycited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025FinMMDocR: Benchmarking Financial Multimodal Reasoning with Scenario Awareness, Document Understanding, and Multi-Step Computation
  2. 2025FinMR: A Knowledge-Intensive Multimodal Benchmark for Advanced Financial Reasoning
  3. 2026FIND: Toward Multimodal Financial Reasoning and Question Answering for Indic Languages
  4. 2025CFBenchmark-MM: Chinese Financial Assistant Benchmark for Multimodal Large Language Model
  5. 2025FinLMM-R1: Enhancing Financial Reasoning in LMM through Scalable Data and Reward Design
  6. 2026FinMTM: A Multi-Turn Multimodal Benchmark for Financial Reasoning and Agent Evaluation
  7. 2024Open-FinLLMs: Open Multimodal Large Language Models for Financial Applications
  8. 2025XFinBench: Benchmarking LLMs in Complex Financial Problem Solving and Reasoning