mlsanityData to 7 Oct 2026

FinanceBench: A New Benchmark for Financial Question Answering

arXiv 2311.11944 ↗First posted 20 Nov 2023cs.CL cs.AI cs.CEcited by 261

Not on the map. Why?

Cited by 63 in LLM agents and tool use, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

FinanceBench is a first-of-its-kind test suite for evaluating the performance of LLMs on open book financial question answering (QA). It comprises 10,231 questions about publicly traded companies, with corresponding answers and evidence strings. The questions in FinanceBench are ecologically valid and cover a diverse set of scenarios. They are intended to be clear-cut and straightforward to answer to serve as a minimum performance standard. We test 16 state of the art model configurations (including GPT-4-Turbo, Llama2 and Claude2, with vector stores and long context prompts) on a sample of 150 cases from FinanceBench, and manually review their answers (n=2,400). The cases are available open-source. We show that existing LLMs have clear limitations for financial QA. Notably, GPT-4-Turbo used with a retrieval system incorrectly answered or refused to answer 81% of questions. While augmentation techniques such as using longer context window to feed in relevant evidence improve performance, they are unrealistic for enterprise settings due to increased latency and cannot support larger financial documents. We find that all models examined exhibit weaknesses, such as hallucinations, that limit their suitability for use by enterprises.

Key facts. Cited by 261 papers in the map (data through 7 Oct 2026). Builds on 21 of its 38 references that are in the map. First posted on arXiv 20 Nov 2023.

Builds on

21 of 38 references.

References · 21

  1. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
  2. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,115
  3. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,018
  4. 2018HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answeringcited by 3,686
  5. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 2,950
  6. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,919
  7. 2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,579
  8. 2018Know What You Don't Know: Unanswerable Questions for SQuADcited by 1,538
  9. 2023Capabilities of GPT-4 on Medical Challenge Problemscited by 630
  10. 2017The NarrativeQA Reading Comprehension Challengecited by 590
  11. 2021FinQA: A Dataset of Numerical Reasoning over Financial Datacited by 552
  12. 2021TAT-QA: A Question Answering Benchmark on a Hybrid of Tabular and Textual Content in Financecited by 402
9 more
  1. 2022Reasoning with Language Model Prompting: A Surveycited by 251
  2. 2022Towards Understanding Chain-of-Thought Prompting: An Empirical Study of What Matterscited by 221
  3. 2022ConvFinQA: Exploring the Chain of Numerical Reasoning in Conversational Finance Question Answeringcited by 194
  4. 2022A Survey of Deep Learning for Mathematical Reasoningcited by 124
  5. 2022WHEN FLUE MEETS FLANG: Benchmarks and Large Pre-trained Language Model for Financial Domaincited by 117
  6. 2023Anchor Points: Benchmarking Models with Much Fewer Examplescited by 84
  7. 2020Towards Ecologically Valid Research on Language User Interfacescited by 42
  8. 2023Can GPT models be Financial Analysts? An Evaluation of ChatGPT and GPT-4 on mock CFA Examscited by 18
  9. 2023Learning to Generalize for Cross-domain QAcited by 6

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 39; 2025: 84; 2026: 138.

261 papers.

Sort:

By area:

  1. 2025Zep: A Temporal Knowledge Graph Architecture for Agent Memorycited by 372
  2. 2024Judging the Judges: Evaluating Alignment and Vulnerabilities in LLMs-as-Judgescited by 197
  3. 2024MMLongBench-Doc: Benchmarking Long-context Document Understanding with Visualizationscited by 174
  4. 2024FinBen: A Holistic Financial Benchmark for Large Language Modelscited by 140
  5. 2024HybridRAG: Integrating Knowledge Graphs and Vector Retrieval Augmented Generation for Efficient Information Extractioncited by 92
  6. 2024A Survey of Large Language Models for Financial Applications: Progress, Prospects and Challengescited by 86
  7. 2025Cartridges: Lightweight and general-purpose long context representations via self-studycited by 74
  8. 2024Introducing v0.5 of the AI Safety Benchmark from MLCommonscited by 60
  9. 2024Lynx: An Open Source Hallucination Evaluation Modelcited by 55
  10. 2024FinTextQA: A Dataset for Long-form Financial Question Answeringcited by 54
  11. 2025MTBench: A Multimodal Time Series Benchmark for Temporal Reasoning and Question Answeringcited by 50
  12. 2024Financial Report Chunking for Effective Retrieval Augmented Generationcited by 46
38 more
  1. 2026OfficeQA Pro: An Enterprise Benchmark for End-to-End Grounded Reasoningcited by 37
  2. 2025REAL-MM-RAG: A Real-World Multi-Modal Retrieval Benchmarkcited by 34
  3. 2025X-MAS: Towards Building Multi-Agent Systems with Heterogeneous LLMscited by 33
  4. 2024FinTral: A Family of GPT-4 Level Multimodal Financial Large Language Modelscited by 32
  5. 2025FinAgentBench: A Benchmark Dataset for Agentic Retrieval in Financial Question Answeringcited by 29
  6. 2025PRBench: Large-Scale Expert Rubrics for Evaluating High-Stakes Professional Reasoningcited by 29
  7. 2024OmniEval: An Omnidirectional and Automatic RAG Evaluation Benchmark in Financial Domaincited by 28
  8. 2025FinMME: Benchmark Dataset for Financial Multi-Modal Reasoning Evaluationcited by 27
  9. 2025Toward Generalizable Evaluation in the LLM Era: A Survey Beyond Benchmarkscited by 26
  10. 2024SV-RAG: LoRA-Contextualizing Adaptation of MLLMs for Long Document Understandingcited by 25
  11. 2024OCR Hinders RAG: Evaluating the Cascading Impact of OCR on Retrieval-Augmented Generationcited by 24
  12. 2025Demystifying Domain-adaptive Post-training for Financial LLMscited by 24
  13. 2024Benchmarking Large Language Models on CFLUE -- A Chinese Financial Language Understanding Evaluation Datasetcited by 23
  14. 2024Long Context RAG Performance of Large Language Modelscited by 23
  15. 2025Agentic Plan Caching: Test-Time Memory for Fast and Cost-Efficient LLM Agentscited by 23
  16. 2025Time Travel is Cheating: Going Live with DeepFund for Real-Time Fund Investment Benchmarkingcited by 21
  17. 2024RAD-Bench: Evaluating Large Language Models Capabilities in Retrieval Augmented Dialoguescited by 19
  18. 2024FAMMA: A Benchmark for Financial Domain Multilingual Multimodal Question Answeringcited by 19
  19. 2024ChemEval: A Comprehensive Multi-Level Chemical Evaluation for Large Language Modelscited by 18
  20. 2025FinMTEB: Finance Massive Text Embedding Benchmarkcited by 18
  21. 2025Minions: Cost-efficient Collaboration Between On-device and Cloud Language Modelscited by 17
  22. 2025FinDER: Financial Dataset for Question Answering and Evaluating Retrieval-Augmented Generationcited by 17
  23. 2025Plutus: Benchmarking Large Language Models in Low-Resource Greek Financecited by 16
  24. 2026FinCARDS: Card-Based Analyst Reranking for Financial Document Question Answeringcited by 16
  25. 2024Mixing It Up: The Cocktail Effect of Multi-Task Fine-Tuning on LLM Performance -- A Case Study in Financecited by 15
  26. 2025Exposing Numeracy Gaps: A Benchmark to Evaluate Fundamental Numerical Abilities in Large Language Modelscited by 14
  27. 2025MultiFinBen: Benchmarking Large Language Models for Multilingual and Multimodal Financial Applicationcited by 14
  28. 2025Standard Benchmarks Fail -- Auditing LLM Agents in Finance Must Prioritize Riskcited by 13
  29. 2025VisR-Bench: An Empirical Study on Visual Retrieval-Augmented Generation for Multilingual Long Document Understandingcited by 13
  30. 2024Do We Need Domain-Specific Embedding Models? An Empirical Investigationcited by 12
  31. 2024Greenback Bears and Fiscal Hawks: Finance is a Jungle and Text Embeddings Must Adaptcited by 12
  32. 2024Enhancing Q&A with Domain-Specific Fine-Tuning and Iterative Reasoning: A Comparative Studycited by 11
  33. 2025Learning Facts at Scale with Active Readingcited by 11
  34. 2024Evaluation-Driven Development and Operations of LLM Agents: A Process Model and Reference Architecturecited by 10
  35. 2025Inferring Events from Time Series using Language Modelscited by 10
  36. 2025Keyword search is all you need: Achieving RAG-Level Performance without vector databases using agentic tool usecited by 10
  37. 2025Optimizing Retrieval Strategies for Financial Question Answering Documents in Retrieval-Augmented Generation Systemscited by 9
  38. 2025Finch: Benchmarking Finance & Accounting across Spreadsheet-Centric Enterprise Workflowscited by 9

50 most cited of 261 All 261 (JSON)