mlsanityData to 7 Oct 2026

RealFin: How Well Do LLMs Reason About Finance When Users Leave Things Unsaid?

arXiv 2602.07096 ↗First posted 6 Feb 2026q-fin.ST cs.AI q-fin.CPcited by 9

Not on the map. Why?

Abstract (arXiv)

Reliable financial reasoning requires knowing not only how to answer, but also when an answer cannot be justified. In real financial practice, problems often rely on implicit assumptions that are taken for granted rather than stated explicitly, causing problems to appear solvable while lacking enough information for a definite answer. We introduce REALFIN, a bilingual benchmark that evaluates financial reasoning by systematically removing essential premises from exam-style questions while keeping them linguistically plausible. Based on this, we evaluate models under three formulations that test answering, recognizing missing information, and rejecting unjustified options, and find consistent performance drops when key conditions are absent. General-purpose models tend to over-commit and guess, while most finance-specialized models fail to clearly identify missing premises. These results highlight a critical gap in current evaluations and show that reliable financial models must know when a question should not be answered.

Key facts. Cited by 9 papers in the map (data through 7 Oct 2026). Builds on 25 of its 42 references that are in the map. First posted on arXiv 6 Feb 2026.

Builds on

25 of 42 references.

References · 25

  1. 2018Know What You Don't Know: Unanswerable Questions for SQuADcited by 1,538
  2. 2023Can LLMs Express Their Uncertainty? An Empirical Evaluation of Confidence Elicitation in LLMscited by 946
  3. 2023BloombergGPT: A Large Language Model for Financecited by 923
  4. 2017Selective Classification for Deep Neural Networkscited by 799
  5. 2021FinQA: A Dataset of Numerical Reasoning over Financial Datacited by 552
  6. 2021TAT-QA: A Question Answering Benchmark on a Hybrid of Tabular and Textual Content in Financecited by 402
  7. 2019FinBERT: Financial Sentiment Analysis with Pre-trained Language Modelscited by 378
  8. 2023FinanceBench: A New Benchmark for Financial Question Answeringcited by 261
  9. 2024A Multimodal Foundation Agent for Financial Trading: Tool-Augmented, Diversified, and Generalistcited by 137
  10. 2025Can MLLMs Reason in Multimodality? EMMA: An Enhanced MultiModal ReAsoning Benchmarkcited by 132
  11. 2025AbstentionBench: Reasoning LLMs Fail on Unanswerable Questionscited by 129
  12. 2023XuanYuan 2.0: A Large Chinese Financial Chat Model with Hundreds of Billions Parameterscited by 92
13 more
  1. 2023DISC-FinLLM: A Chinese Financial Large Language Model based on Multiple Experts Fine-tuningcited by 64
  2. 2025QuestBench: Can LLMs ask the right question to acquire information in reasoning tasks?cited by 56
  3. 2024Do LLMs Know When to NOT Answer? Investigating Abstention Abilities of Large Language Modelscited by 55
  4. 2024Characterizing LLM Abstention Behavior in Science QA with Context Perturbationscited by 26
  5. 2023CFGPT: Chinese Financial Assistant with Large Language Modelcited by 11
  6. 2025Beyond the Black Box: Interpretability of LLMs in Financecited by 10
  7. 2025FinMaster: A Holistic Benchmark for Mastering Full-Pipeline Financial Workflows with LLMscited by 9
  8. 2024AI as Decision-Maker: Ethics and Risk Preferences of LLMscited by 7
  9. 2025Open FinLLM Leaderboard: Towards Financial AI Readinesscited by 6
  10. 2024Forecasting Credit Ratings: A Case Study where Traditional Methods Outperform Generative LLMscited by 5
  11. 2024Chat Bankman-Fried: an Exploration of LLM Alignment in Financecited by 5
  12. 2025FinTeam: A Multi-Agent Collaborative Intelligence System for Comprehensive Financial Scenarioscited by 5
  13. 2025Extract, Match, and Score: An Evaluation Paradigm for Long Question-context-answer Triplets in Financial Analysiscited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 9.

9 papers.