# RealFin: How Well Do LLMs Reason About Finance When Users Leave Things Unsaid?

arXiv: https://arxiv.org/abs/2602.07096 · HTML: https://mlsanity.com/paper/2602.07096/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Builds on 25 of its 42 references that are in the map.
- First posted on arXiv 6 Feb 2026.

## Builds on (most cited first, 25 of 25)

- [Know What You Don't Know: Unanswerable Questions for SQuAD](https://mlsanity.com/paper/1806.03822/) (2018)
- [Can LLMs Express Their Uncertainty? An Empirical Evaluation of Confidence Elicitation in LLMs](https://mlsanity.com/paper/2306.13063/) (2023)
- [BloombergGPT: A Large Language Model for Finance](https://mlsanity.com/paper/2303.17564/) (2023)
- [Selective Classification for Deep Neural Networks](https://mlsanity.com/paper/1705.08500/) (2017)
- [FinQA: A Dataset of Numerical Reasoning over Financial Data](https://mlsanity.com/paper/2109.00122/) (2021)
- [TAT-QA: A Question Answering Benchmark on a Hybrid of Tabular and Textual Content in Finance](https://mlsanity.com/paper/2105.07624/) (2021)
- [FinBERT: Financial Sentiment Analysis with Pre-trained Language Models](https://mlsanity.com/paper/1908.10063/) (2019)
- [FinanceBench: A New Benchmark for Financial Question Answering](https://mlsanity.com/paper/2311.11944/) (2023)
- [AbstentionBench: Reasoning LLMs Fail on Unanswerable Questions](https://mlsanity.com/paper/2506.09038/) (2025)
- [A Multimodal Foundation Agent for Financial Trading: Tool-Augmented, Diversified, and Generalist](https://mlsanity.com/paper/2402.18485/) (2024)
- [Can MLLMs Reason in Multimodality? EMMA: An Enhanced MultiModal ReAsoning Benchmark](https://mlsanity.com/paper/2501.05444/) (2025)
- [XuanYuan 2.0: A Large Chinese Financial Chat Model with Hundreds of Billions Parameters](https://mlsanity.com/paper/2305.12002/) (2023)
- [DISC-FinLLM: A Chinese Financial Large Language Model based on Multiple Experts Fine-tuning](https://mlsanity.com/paper/2310.15205/) (2023)
- [QuestBench: Can LLMs ask the right question to acquire information in reasoning tasks?](https://mlsanity.com/paper/2503.22674/) (2025)
- [Do LLMs Know When to NOT Answer? Investigating Abstention Abilities of Large Language Models](https://mlsanity.com/paper/2407.16221/) (2024)
- [Characterizing LLM Abstention Behavior in Science QA with Context Perturbations](https://mlsanity.com/paper/2404.12452/) (2024)
- [CFGPT: Chinese Financial Assistant with Large Language Model](https://mlsanity.com/paper/2309.10654/) (2023)
- [Beyond the Black Box: Interpretability of LLMs in Finance](https://mlsanity.com/paper/2505.24650/) (2025)
- [FinMaster: A Holistic Benchmark for Mastering Full-Pipeline Financial Workflows with LLMs](https://mlsanity.com/paper/2505.13533/) (2025)
- [AI as Decision-Maker: Ethics and Risk Preferences of LLMs](https://mlsanity.com/paper/2406.01168/) (2024)
- [Open FinLLM Leaderboard: Towards Financial AI Readiness](https://mlsanity.com/paper/2501.10963/) (2025)
- [Forecasting Credit Ratings: A Case Study where Traditional Methods Outperform Generative LLMs](https://mlsanity.com/paper/2407.17624/) (2024)
- [Chat Bankman-Fried: an Exploration of LLM Alignment in Finance](https://mlsanity.com/paper/2411.11853/) (2024)
- [FinTeam: A Multi-Agent Collaborative Intelligence System for Comprehensive Financial Scenarios](https://mlsanity.com/paper/2507.10448/) (2025)
- [Extract, Match, and Score: An Evaluation Paradigm for Long Question-context-answer Triplets in Financial Analysis](https://mlsanity.com/paper/2503.16575/) (2025)

## Built on by (most cited first, 9 of 9)

- [GenericAgent: A Token-Efficient Self-Evolving LLM Agent via Contextual Information Density Maximization (V1.0)](https://mlsanity.com/paper/2604.17091/) (2026)
- [The CLEF-2026 FinMMEval Lab: Multilingual and Multimodal Evaluation of Financial AI Systems](https://mlsanity.com/paper/2602.10886/) (2026)
- [Bayesian-Agent: Posterior-Guided Skill Evolution Across LLM Agent Harnesses](https://mlsanity.com/paper/2606.08348/) (2026)
- [Overview of FinMMEval 2026 Task 1: Multilingual Financial Multiple-Choice Question Answering](https://mlsanity.com/paper/2607.19856/) (2026)
- [Overview of FinMMEval 2026 Task 2: Multilingual Financial Short-Answer Question Answering](https://mlsanity.com/paper/2607.19867/) (2026)
- [Seeing Is Not Deciding: Can Multimodal LLMs Act as Effective CEOs?](https://mlsanity.com/paper/2608.05864/) (2026)
- [FinRiskAtlas: Decision-Aligned Evaluation of Large Language Models for Financial Risk Review](https://mlsanity.com/paper/2608.25325/) (2026)
- [FinExam-10K: When Retrieval Helps Financial Reasoning?](https://mlsanity.com/paper/2608.28155/) (2026)
- [Solving versus Verifying: Catching Contradictions in Tax Reasoning Systems](https://mlsanity.com/paper/2609.05928/) (2026)

## Abstract (arXiv)

Reliable financial reasoning requires knowing not only how to answer, but also when an answer cannot be justified. In real financial practice, problems often rely on implicit assumptions that are taken for granted rather than stated explicitly, causing problems to appear solvable while lacking enough information for a definite answer. We introduce REALFIN, a bilingual benchmark that evaluates financial reasoning by systematically removing essential premises from exam-style questions while keeping them linguistically plausible. Based on this, we evaluate models under three formulations that test answering, recognizing missing information, and rejecting unjustified options, and find consistent performance drops when key conditions are absent. General-purpose models tend to over-commit and guess, while most finance-specialized models fail to clearly identify missing premises. These results highlight a critical gap in current evaluations and show that reliable financial models must know when a question should not be answered.