mlsanityData to 14 Oct 2026

Finance Language Model Evaluation (FLaME)

arXiv 2506.15846 ↗First posted 18 Jun 2025cs.CL cs.AI cs.CEcited by 4

Not on the map. Why?

Abstract (arXiv)

Language Models (LMs) have demonstrated impressive capabilities with core Natural Language Processing (NLP) tasks. The effectiveness of LMs for highly specialized knowledge-intensive tasks in finance remains difficult to assess due to major gaps in the methodologies of existing evaluation frameworks, which have caused an erroneous belief in a far lower bound of LMs' performance on common Finance NLP (FinNLP) tasks. To demonstrate the potential of LMs for these FinNLP tasks, we present the first holistic benchmarking suite for Financial Language Model Evaluation (FLaME). We are the first research paper to comprehensively study LMs against 'reasoning-reinforced' LMs, with an empirical study of 23 foundation LMs over 20 core NLP tasks in finance. We open-source our framework software along with all data and results.

Key facts. Cited by 4 papers in the map (data through 14 Oct 2026). Builds on 46 of its 97 references that are in the map. First posted on arXiv 18 Jun 2025.

Builds on

46 of 97 references.

References · 46

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  3. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,899
  4. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,886
  5. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,148
  6. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,100
  7. 2021On the Opportunities and Risks of Foundation Modelscited by 4,033
  8. 2022Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language modelscited by 1,998
  9. 2024MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmarkcited by 1,848
  10. 2022Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Themcited by 1,807
  11. 2019SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systemscited by 1,729
  12. 2023BloombergGPT: A Large Language Model for Financecited by 939
34 more
  1. 2021FinQA: A Dataset of Numerical Reasoning over Financial Datacited by 562
  2. 2021Dynabench: Rethinking Benchmarking in NLPcited by 459
  3. 2020Efficient Intent Detection with Dual Sentence Encoderscited by 433
  4. 2021TAT-QA: A Question Answering Benchmark on a Hybrid of Tabular and Textual Content in Financecited by 408
  5. 2021AI and the Everything in the Whole Wide World Benchmarkcited by 349
  6. 2022Demonstrate-Search-Predict: Composing retrieval and language models for knowledge-intensive NLPcited by 318
  7. 2013Good Debt or Bad Debt: Detecting Semantic Orientations in Economic Textscited by 310
  8. 2024Optimizing Instructions and Demonstrations for Multi-Stage Language Model Programscited by 291
  9. 2023FinanceBench: A New Benchmark for Financial Question Answeringcited by 270
  10. 2023PIXIU: A Large Language Model, Instruction Data and Evaluation Benchmark for Financecited by 222
  11. 2023ChatGPT: Jack of all trades, master of nonecited by 204
  12. 2021How Many Data Points is a Prompt Worth?cited by 198
  13. 2022ConvFinQA: Exploring the Chain of Numerical Reasoning in Conversational Finance Question Answeringcited by 196
  14. 2021The Values Encoded in Machine Learning Researchcited by 172
  15. 2022On Decoding Strategies for Neural Text Generatorscited by 90
  16. 2024Revolutionizing Finance with LLMs: An Overview of Applications and Insightscited by 90
  17. 2024A Survey of Large Language Models for Financial Applications: Progress, Prospects and Challengescited by 90
  18. 2020Impact of News on the Commodity Market: Dataset and Resultscited by 80
  19. 2023Trillion Dollar Words: A New Financial Dataset, Task & Market Analysiscited by 75
  20. 2024A Survey on Large Language Models for Critical Societal Domains: Finance, Healthcare, and Lawcited by 74
  21. 2023The Data Provenance Initiative: A Large Scale Audit of Dataset Licensing & Attribution in AIcited by 73
  22. 2022ECTSum: A New Benchmark Dataset For Bullet Point Summarization of Long Earnings Call Transcriptscited by 59
  23. 2021Dynaboard: An Evaluation-As-A-Service Platform for Holistic Next-Generation Benchmarkingcited by 53
  24. 2024Designing Heterogeneous LLM Agents for Financial Sentiment Analysiscited by 48
  25. 2023Deficiency of Large Language Models in Finance: An Empirical Examination of Hallucinationcited by 41
  26. 2021Trade the Event: Corporate Events Detection for News-Based Event-Driven Tradingcited by 39
  27. 2020Financial Document Causality Detection Shared Task (FinCausal 2020)cited by 31
  28. 2023Beyond Prompts: Exploring the Design Space of Mixed-Initiative Co-Creativity Systemscited by 20
  29. 2023FinEntity: Entity-level Sentiment Classification for Financial Textscited by 14
  30. 2023FinPT: Financial Risk Prediction with Profile Tuning on Pretrained Foundation Modelscited by 13
  31. 2023Financial Numeric Extreme Labelling: A Dataset and Benchmarking for XBRL Taggingcited by 12
  32. 2024SubjECTive-QA: Measuring Subjectivity in Earnings Call Transcripts' QA Through Six-Dimensional Feature Analysiscited by 8
  33. 2025Financial Named Entity Recognition: How Far Can LLM Go?cited by 8
  34. 2023An Ontology of Co-Creative AI Systemscited by 7

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 2; 2026: 2.

4 papers.

  1. 2026FINESSE-Bench: A Hierarchical Benchmark Suite for Financial Domain Knowledge and Technical Analysis in Large Language Modelscited by 4
  2. 2025Synthesizing Behaviorally-Grounded Reasoning Chains: A Data-Generation Framework for Personal Finance LLMscited by 1
  3. 2025Financial Instruction Following Evaluation (FIFE)cited by 0
  4. 2026RusFinChain: A Russian Benchmark for Verifiable Chain-of-Thought Reasoning in Finance with Fuzzy-Aligned Evaluationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024FinBen: A Holistic Financial Benchmark for Large Language Models
  2. 2026FinForge: Semi-Synthetic Financial Benchmark Generation
  3. 2025Evaluating Large Language Models (LLMs) in Financial NLP: A Comparative Study on Financial Report Analysis
  4. 2025FINCH: Financial Intelligence using Natural language for Contextualized SQL Handling
  5. 2026Are Large Language Models Reliable Reviewers? A Benchmark for Error Detection in Financial Documents
  6. 2025FinNLI: Novel Dataset for Multi-Genre Financial Natural Language Inference Benchmarking
  7. 2023Beyond Classification: Financial Reasoning in State-of-the-Art Language Models
  8. 2026FinMCP-Bench: Benchmarking LLM Agents for Real-World Financial Tool Use under the Model Context Protocol