RealFin: How Well Do LLMs Reason About Finance When Users Leave Things Unsaid?
Not on the map. Why?
Abstract (arXiv)
Reliable financial reasoning requires knowing not only how to answer, but also when an answer cannot be justified. In real financial practice, problems often rely on implicit assumptions that are taken for granted rather than stated explicitly, causing problems to appear solvable while lacking enough information for a definite answer. We introduce REALFIN, a bilingual benchmark that evaluates financial reasoning by systematically removing essential premises from exam-style questions while keeping them linguistically plausible. Based on this, we evaluate models under three formulations that test answering, recognizing missing information, and rejecting unjustified options, and find consistent performance drops when key conditions are absent. General-purpose models tend to over-commit and guess, while most finance-specialized models fail to clearly identify missing premises. These results highlight a critical gap in current evaluations and show that reliable financial models must know when a question should not be answered.
Key facts. Cited by 9 papers in the map (data through 7 Oct 2026). Builds on 25 of its 42 references that are in the map. First posted on arXiv 6 Feb 2026.
Builds on
25 of 42 references.
References · 25
- 2018Know What You Don't Know: Unanswerable Questions for SQuADcited by 1,538
- 2023Can LLMs Express Their Uncertainty? An Empirical Evaluation of Confidence Elicitation in LLMscited by 946
- 2023BloombergGPT: A Large Language Model for Financecited by 923
- 2017Selective Classification for Deep Neural Networkscited by 799
- 2021FinQA: A Dataset of Numerical Reasoning over Financial Datacited by 552
- 2021TAT-QA: A Question Answering Benchmark on a Hybrid of Tabular and Textual Content in Financecited by 402
- 2019FinBERT: Financial Sentiment Analysis with Pre-trained Language Modelscited by 378
- 2023FinanceBench: A New Benchmark for Financial Question Answeringcited by 261
- 2024A Multimodal Foundation Agent for Financial Trading: Tool-Augmented, Diversified, and Generalistcited by 137
- 2025Can MLLMs Reason in Multimodality? EMMA: An Enhanced MultiModal ReAsoning Benchmarkcited by 132
- 2025AbstentionBench: Reasoning LLMs Fail on Unanswerable Questionscited by 129
- 2023XuanYuan 2.0: A Large Chinese Financial Chat Model with Hundreds of Billions Parameterscited by 92
13 more
- 2023DISC-FinLLM: A Chinese Financial Large Language Model based on Multiple Experts Fine-tuningcited by 64
- 2025QuestBench: Can LLMs ask the right question to acquire information in reasoning tasks?cited by 56
- 2024Do LLMs Know When to NOT Answer? Investigating Abstention Abilities of Large Language Modelscited by 55
- 2024Characterizing LLM Abstention Behavior in Science QA with Context Perturbationscited by 26
- 2023CFGPT: Chinese Financial Assistant with Large Language Modelcited by 11
- 2025Beyond the Black Box: Interpretability of LLMs in Financecited by 10
- 2025FinMaster: A Holistic Benchmark for Mastering Full-Pipeline Financial Workflows with LLMscited by 9
- 2024AI as Decision-Maker: Ethics and Risk Preferences of LLMscited by 7
- 2025Open FinLLM Leaderboard: Towards Financial AI Readinesscited by 6
- 2024Forecasting Credit Ratings: A Case Study where Traditional Methods Outperform Generative LLMscited by 5
- 2024Chat Bankman-Fried: an Exploration of LLM Alignment in Financecited by 5
- 2025FinTeam: A Multi-Agent Collaborative Intelligence System for Comprehensive Financial Scenarioscited by 5
- 2025Extract, Match, and Score: An Evaluation Paradigm for Long Question-context-answer Triplets in Financial Analysiscited by 3
Built on by
Citing papers per year: 2026: 9.
9 papers.
- 2026GenericAgent: A Token-Efficient Self-Evolving LLM Agent via Contextual Information Density Maximization (V1.0)cited by 10
- 2026The CLEF-2026 FinMMEval Lab: Multilingual and Multimodal Evaluation of Financial AI Systemscited by 2
- 2026Bayesian-Agent: Posterior-Guided Skill Evolution Across LLM Agent Harnessescited by 0
- 2026Overview of FinMMEval 2026 Task 1: Multilingual Financial Multiple-Choice Question Answeringcited by 0
- 2026Overview of FinMMEval 2026 Task 2: Multilingual Financial Short-Answer Question Answeringcited by 0
- 2026Seeing Is Not Deciding: Can Multimodal LLMs Act as Effective CEOs?cited by 0
- 2026FinRiskAtlas: Decision-Aligned Evaluation of Large Language Models for Financial Risk Reviewcited by 0
- 2026FinExam-10K: When Retrieval Helps Financial Reasoning?cited by 0
- 2026Solving versus Verifying: Catching Contradictions in Tax Reasoning Systemscited by 0
- 2026Solving versus Verifying: Catching Contradictions in Tax Reasoning Systems5 Sep 2026
- 2026FinExam-10K: When Retrieval Helps Financial Reasoning?28 Aug 2026
- 2026FinRiskAtlas: Decision-Aligned Evaluation of Large Language Models for Financial Risk Review26 Aug 2026
- 2026Seeing Is Not Deciding: Can Multimodal LLMs Act as Effective CEOs?6 Aug 2026
- 2026Overview of FinMMEval 2026 Task 2: Multilingual Financial Short-Answer Question Answering22 Jul 2026
- 2026Overview of FinMMEval 2026 Task 1: Multilingual Financial Multiple-Choice Question Answering22 Jul 2026
- 2026Bayesian-Agent: Posterior-Guided Skill Evolution Across LLM Agent Harnesses6 Jun 2026
- 2026GenericAgent: A Token-Efficient Self-Evolving LLM Agent via Contextual Information Density Maximization (V1.0)18 Apr 2026
- 2026The CLEF-2026 FinMMEval Lab: Multilingual and Multimodal Evaluation of Financial AI Systems11 Feb 2026