mlsanityData to 7 Oct 2026

FinBen: A Holistic Financial Benchmark for Large Language Models

arXiv 2402.12659 ↗First posted 20 Feb 2024cs.CL cs.AI cs.CEcited by 140

Not on the map. Why?

Cited by 28 in LLM agents and tool use, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

LLMs have transformed NLP and shown promise in various fields, yet their potential in finance is underexplored due to a lack of comprehensive evaluation benchmarks, the rapid development of LLMs, and the complexity of financial tasks. In this paper, we introduce FinBen, the first extensive open-source evaluation benchmark, including 36 datasets spanning 24 financial tasks, covering seven critical aspects: information extraction (IE), textual analysis, question answering (QA), text generation, risk management, forecasting, and decision-making. FinBen offers several key innovations: a broader range of tasks and datasets, the first evaluation of stock trading, novel agent and Retrieval-Augmented Generation (RAG) evaluation, and three novel open-source evaluation datasets for text summarization, question answering, and stock trading. Our evaluation of 15 representative LLMs, including GPT-4, ChatGPT, and the latest Gemini, reveals several key findings: While LLMs excel in IE and textual analysis, they struggle with advanced reasoning and complex tasks like text generation and forecasting. GPT-4 excels in IE and stock trading, while Gemini is better at text generation and forecasting. Instruction-tuned LLMs improve textual analysis but offer limited benefits for complex tasks such as QA. FinBen has been used to host the first financial LLMs shared task at the FinNLP-AgentScen workshop during IJCAI-2024, attracting 12 teams. Their novel solutions outperformed GPT-4, showcasing FinBen's potential to drive innovation in financial LLMs. All datasets, results, and codes are released for the research community: https://github.com/The-FinAI/PIXIU.

Key facts. Cited by 140 papers in the map (data through 7 Oct 2026). Builds on 45 of its 99 references that are in the map. First posted on arXiv 20 Feb 2024.

Builds on

45 of 99 references.

References · 45

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  4. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,462
  5. 2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,579
  6. 2023Code Llama: Open Foundation Models for Codecited by 2,105
  7. 2021GLM: General Language Model Pretraining with Autoregressive Blank Infillingcited by 1,026
  8. 2023BloombergGPT: A Large Language Model for Financecited by 923
  9. 2021BARTScore: Evaluating Generated Text as Text Generationcited by 719
  10. 2023Baichuan 2: Open Large-scale Language Modelscited by 664
  11. 2023Instruction Tuning for Large Language Models: A Surveycited by 617
  12. 2021FinQA: A Dataset of Numerical Reasoning over Financial Datacited by 552
33 more
  1. 2023The Falcon Series of Open Language Modelscited by 526
  2. 2021TAT-QA: A Question Answering Benchmark on a Hybrid of Tabular and Textual Content in Financecited by 402
  3. 2019FinBERT: Financial Sentiment Analysis with Pre-trained Language Modelscited by 378
  4. 2013Good Debt or Bad Debt: Detecting Semantic Orientations in Economic Textscited by 310
  5. 2023FinanceBench: A New Benchmark for Financial Question Answeringcited by 261
  6. 2023Large Language Models in Finance: A Surveycited by 223
  7. 2023PIXIU: A Large Language Model, Instruction Data and Evaluation Benchmark for Financecited by 217
  8. 2022ConvFinQA: Exploring the Chain of Numerical Reasoning in Conversational Finance Question Answeringcited by 194
  9. 2023FinMem: A Performance-Enhanced LLM Trading Agent with Layered Memory and Character Designcited by 163
  10. 2020FinBERT: A Pretrained Language Model for Financial Communicationscited by 161
  11. 2022WHEN FLUE MEETS FLANG: Benchmarks and Large Pre-trained Language Model for Financial Domaincited by 117
  12. 2023XuanYuan 2.0: A Large Chinese Financial Chat Model with Hundreds of Billions Parameterscited by 92
  13. 2024Revolutionizing Finance with LLMs: An Overview of Applications and Insightscited by 88
  14. 2021AI in Finance: Challenges, Techniques and Opportunitiescited by 81
  15. 2020Impact of News on the Commodity Market: Dataset and Resultscited by 80
  16. 2023InvestLM: A Large Language Model for Investment using Financial Domain Instruction Tuningcited by 80
  17. 2024A Survey of Large Language Models in Finance (FinLLMs)cited by 77
  18. 2023Trillion Dollar Words: A New Financial Dataset, Task & Market Analysiscited by 73
  19. 2023FinGPT: Instruction Tuning Benchmark for Open-Source Large Language Models in Financial Datasetscited by 68
  20. 2023DISC-FinLLM: A Chinese Financial Large Language Model based on Multiple Experts Fine-tuningcited by 64
  21. 2022ECTSum: A New Benchmark Dataset For Bullet Point Summarization of Long Earnings Call Transcriptscited by 59
  22. 2023The Wall Street Neophyte: A Zero-Shot Analysis of ChatGPT Over MultiModal Stock Movement Prediction Challengescited by 59
  23. 2023DocMath-Eval: Evaluating Math Reasoning Capabilities of LLMs in Understanding Long and Specialized Documentscited by 45
  24. 2020Generating Plausible Counterfactual Explanations for Deep Transformers in Financial Text Classificationcited by 40
  25. 2021Trade the Event: Corporate Events Detection for News-Based Event-Driven Tradingcited by 39
  26. 2020Financial Document Causality Detection Shared Task (FinCausal 2020)cited by 31
  27. 2023BizBench: A Quantitative Reasoning Benchmark for Business and Financecited by 28
  28. 2023Dynamic Datasets and Market Environments for Financial Reinforcement Learningcited by 15
  29. 2023Financial Numeric Extreme Labelling: A Dataset and Benchmarking for XBRL Taggingcited by 11
  30. 2023CGCE: A Chinese Generative Chat Evaluation Benchmark for General and Financial Domainscited by 7
  31. 2018Textual Analogy Parsing: What's Shared and What's Compared among Analogous Factscited by 5
  32. 2023Mastering Pair Trading with Risk-Aware Recurrent Reinforcement Learningcited by 3
  33. 2023How are We Detecting Inconsistent Method Names? An Empirical Study from Code Review Perspectivecited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 17; 2025: 53; 2026: 70.

140 papers.

Sort:

By area:

  1. 2024FinCon: A Synthesized LLM Multi-Agent System with Conceptual Verbal Reinforcement for Enhanced Financial Decision Makingcited by 112
  2. 2024A Survey of Large Language Models for Financial Applications: Progress, Prospects and Challengescited by 86
  3. 2025MTBench: A Multimodal Time Series Benchmark for Temporal Reasoning and Question Answeringcited by 50
  4. 2024Peer Review as A Multi-Turn and Long-Context Dialogue with Role-Based Interactionscited by 43
  5. 2024Open-FinLLMs: Open Multimodal Large Language Models for Financial Applicationscited by 36
  6. 2025Fino1: On the Transferability of Reasoning-Enhanced LLMs and Reinforcement Learning to Financecited by 32
  7. 2025FinMME: Benchmark Dataset for Financial Multi-Modal Reasoning Evaluationcited by 27
  8. 2025Toward Generalizable Evaluation in the LLM Era: A Survey Beyond Benchmarkscited by 26
  9. 2025Time Travel is Cheating: Going Live with DeepFund for Real-Time Fund Investment Benchmarkingcited by 21
  10. 2025A Survey of Reasoning and Agentic Systems in Time Series with Large Language Modelscited by 21
  11. 2024FAMMA: A Benchmark for Financial Domain Multilingual Multimodal Question Answeringcited by 19
  12. 2024Can Large Language Models Analyze Graphs like Professionals? A Benchmark, Datasets and Modelscited by 17
38 more
  1. 2024Mixing It Up: The Cocktail Effect of Multi-Task Fine-Tuning on LLM Performance -- A Case Study in Financecited by 15
  2. 2025Standard Benchmarks Fail -- Auditing LLM Agents in Finance Must Prioritize Riskcited by 13
  3. 2024CFinBench: A Comprehensive Chinese Financial Benchmark for Large Language Modelscited by 12
  4. 2025Retrieval-augmented Large Language Models for Financial Time Series Forecastingcited by 12
  5. 2025FinLoRA: Benchmarking LoRA Methods for Fine-Tuning LLMs on Financial Datasetscited by 11
  6. 2025Finch: Benchmarking Finance & Accounting across Spreadsheet-Centric Enterprise Workflowscited by 9
  7. 2025FLAME: Financial Large-Language Model Assessment and Metrics Evaluationcited by 8
  8. 2025LLM-based HSE Compliance Assessment: Benchmark, Performance, and Advancementscited by 8
  9. 2024Revolutionizing Database Q&A with Large Language Models: Comprehensive Benchmark and Evaluationcited by 7
  10. 2024SusGen-GPT: A Data-Centric LLM for Financial NLP and Sustainability Report Generationcited by 7
  11. 2025Expect the Unexpected: FailSafe Long Context QA for Financecited by 7
  12. 2026Agentic Trading: When LLM Agents Meet Financial Marketscited by 7
  13. 2024Evaluating Large Language Models for Public Health Classification and Extraction Taskscited by 6
  14. 2024Multi-Document Financial Question Answering using LLMscited by 6
  15. 2025EDINET-Bench: Evaluating LLMs on Complex Financial Tasks using Japanese Financial Statementscited by 6
  16. 2026Benchmarking Multi-Agent LLM Architectures for Financial Document Processing: A Comparative Study of Orchestration Patterns, Cost-Accuracy Tradeoffs and Production Scaling Strategiescited by 6
  17. 2026BigFinanceBench: A Workflow-Grounded Benchmark for Financial-Research Agentscited by 6
  18. 2025MoodAngels: A Retrieval-augmented Multi-agent Framework for Psychiatry Diagnosiscited by 5
  19. 2025Interpretable LLMs for Credit Risk: A Systematic Review and Taxonomycited by 5
  20. 2025Enterprise Large Language Model Evaluation Benchmarkcited by 5
  21. 2026TAB-PO: Preference Optimization with a Token-Level Adaptive Barrier for Token-Critical Structured Generationcited by 5
  22. 2026ASDA: Automated Skill Distillation and Adaptation for Financial Reasoningcited by 5
  23. 2024RiskLabs: Predicting Financial Risk Using Large Language Model based on Multimodal and Multi-Sources Datacited by 4
  24. 2024Golden Touchstone: A Comprehensive Bilingual Benchmark for Evaluating Financial Large Language Modelscited by 4
  25. 2025Will LLMs be Professional at Fund Investment? DeepFund: A Live Arena Perspectivecited by 4
  26. 2025Advancing Financial Engineering with Foundation Models: Progress, Applications, and Challengescited by 4
  27. 2026Adversarial News and Lost Profits: Manipulating Headlines in LLM-Driven Algorithmic Tradingcited by 4
  28. 2026TraderBench: How Robust Are AI Agents in Adversarial Capital Markets?cited by 4
  29. 2026FinRetrieval: A Benchmark for Financial Data Retrieval by AI Agentscited by 4
  30. 2026From Feedback Loops to Policy Updates: Reinforcement Fine-Tuning for LLM-Based Alpha Factor Discoverycited by 4
  31. 2026FINESSE-Bench: A Hierarchical Benchmark Suite for Financial Domain Knowledge and Technical Analysis in Large Language Modelscited by 4
  32. 2026MBABench: Evaluating LLM Agents on End-to-End Spreadsheet Tasks in Financecited by 4
  33. 2025Learnware of Language Models: Specialized Small Language Models Can Do Bigcited by 3
  34. 2025Large Language Models are Near-Optimal Decision-Makers with a Non-Human Learning Behaviorcited by 3
  35. 2025MASCA: LLM based-Multi Agents System for Credit Assessmentcited by 3
  36. 2026FinTexTS: Financial Text-Paired Time-Series Dataset via Semantic-Based and Multi-Level Pairingcited by 3
  37. 2026Unlocking Data Value in Finance: A Study on Distillation and Difficulty-Aware Trainingcited by 3
  38. 2026IndustryBench: Probing the Industrial Knowledge Boundaries of LLMscited by 3

50 most cited of 140 All 140 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026FinTradeBench: A Financial Reasoning Benchmark for LLMs
  2. 2026FinForge: Semi-Synthetic Financial Benchmark Generation
  3. 2025BizFinBench: A Business-Driven Real-World Financial Benchmark for Evaluating LLMs
  4. 2025Finance Language Model Evaluation (FLaME)
  5. 2026FinToolBench: Evaluating LLM Agents for Real-World Financial Tool Use
  6. 2026FinSheet-Bench: From Simple Lookups to Complex Reasoning, Where LLMs Break on Financial Spreadsheets
  7. 2024INVESTORBENCH: A Benchmark for Financial Decision-Making Tasks with LLM-based Agent
  8. 2025FinS-Pilot: A Benchmark for Online Financial RAG System