mlsanityData to 14 Oct 2026

Can GPT models be Financial Analysts? An Evaluation of ChatGPT and GPT-4 on mock CFA Exams

arXiv 2310.08678 ↗First posted 12 Oct 2023cs.CL cs.AI q-fin.GNcited by 18

Not on the map. Why?

Abstract (arXiv)

Large Language Models (LLMs) have demonstrated remarkable performance on a wide range of Natural Language Processing (NLP) tasks, often matching or even beating state-of-the-art task-specific models. This study aims at assessing the financial reasoning capabilities of LLMs. We leverage mock exam questions of the Chartered Financial Analyst (CFA) Program to conduct a comprehensive evaluation of ChatGPT and GPT-4 in financial analysis, considering Zero-Shot (ZS), Chain-of-Thought (CoT), and Few-Shot (FS) scenarios. We present an in-depth analysis of the models' performance and limitations, and estimate whether they would have a chance at passing the CFA exams. Finally, we outline insights into potential strategies and improvements to enhance the applicability of LLMs in finance. In this perspective, we hope this work paves the way for future studies to continue enhancing LLMs for financial reasoning through rigorous evaluation.

Key facts. Cited by 18 papers in the map (data through 14 Oct 2026). Builds on 21 of its 53 references that are in the map. First posted on arXiv 12 Oct 2023.

Builds on

21 of 53 references.

References · 21

  1. 2017Attention Is All You Needcited by 60,361
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  4. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  5. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,297
  6. 2022Large Language Models are Zero-Shot Reasonerscited by 4,374
  7. 2023BloombergGPT: A Large Language Model for Financecited by 939
  8. 2023A Multitask, Multilingual, Multimodal Evaluation of ChatGPT on Reasoning, Hallucination, and Interactivitycited by 894
  9. 2020UnifiedQA: Crossing Format Boundaries With a Single QA Systemcited by 476
  10. 2021OCR-free Document Understanding Transformercited by 407
  11. 2019FinBERT: Financial Sentiment Analysis with Pre-trained Language Modelscited by 385
  12. 2022CodeRL: Mastering Code Generation through Pretrained Models and Deep Reinforcement Learningcited by 358
9 more
  1. 2023Causal Reasoning and Large Language Models: Opening a New Frontier for Causalitycited by 353
  2. 2023Mathematical Capabilities of ChatGPTcited by 268
  3. 2023SciBench: Evaluating College-Level Scientific Problem-Solving Abilities of Large Language Modelscited by 256
  4. 2022ConvFinQA: Exploring the Chain of Numerical Reasoning in Conversational Finance Question Answeringcited by 196
  5. 2023Few-shot Fine-tuning vs. In-context Learning: A Fair Comparison and Evaluationcited by 136
  6. 2021Unified Named Entity Recognition as Word-Word Relation Classificationcited by 90
  7. 2023Extractive Summarization via ChatGPT for Faithful Summary Generationcited by 59
  8. 2022A Numerical Reasoning Question Answering System with Fine-grained Retriever and the Ensemble of Multiple Generators for FinQAcited by 4
  9. 2023Exploring the Effectiveness of GPT Models in Test-Taking: A Case Study of the Driver's License Knowledge Testcited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 2; 2024: 7; 2025: 8; 2026: 1.

18 papers.

Sort:

By area:

  1. 2023FinanceBench: A New Benchmark for Financial Question Answeringcited by 270
  2. 2023DocMath-Eval: Evaluating Math Reasoning Capabilities of LLMs in Understanding Long and Specialized Documentscited by 45
  3. 2024Can Large Language Models Beat Wall Street? Unveiling the Potential of AI in Stock Selectioncited by 29
  4. 2025Demystifying Domain-adaptive Post-training for Financial LLMscited by 24
  5. 2025FinLoRA: Benchmarking LoRA Methods for Fine-Tuning LLMs on Financial Datasetscited by 12
  6. 2025Evaluating Large Language Models as Expert Annotatorscited by 9
  7. 2024EconNLI: Evaluating Large Language Models on Economics Reasoningcited by 8
  8. 2024Adversarial Math Word Problem Generationcited by 6
  9. 2025Advanced Financial Reasoning at Scale: A Comprehensive Evaluation of Large Language Models on CFA Level IIIcited by 6
  10. 2026FinTexTS: Financial Text-Paired Time-Series Dataset via Semantic-Based and Multi-Level Pairingcited by 5
  11. 2024Financial Knowledge Large Language Modelcited by 3
  12. 2024Are Expert-Level Language Models Expert-Level Annotators?cited by 3
6 more
  1. 2024Quantifying Qualitative Insights: Leveraging LLMs to Market Predictcited by 3
  2. 2025Evaluating Large Language Models for Financial Reasoning: A CFA-Based Benchmark Studycited by 3
  3. 2024THaLLE: Text Hyperlocally Augmented Large Language Extension -- Technical Reportcited by 1
  4. 2025AI Analyst: Framework and Comprehensive Evaluation of Large Language Models for Financial Time Series Report Generationcited by 1
  5. 2025KAHAN: Knowledge-Augmented Hierarchical Analysis and Narration for Financial Data Narrationcited by 1
  6. 2025Reasoning Models Ace the CFA Examscited by 1

Similar, not cited

Close in wording, no citation either way

  1. 2023Beyond Classification: Financial Reasoning in State-of-the-Art Language Models
  2. 2024Can ChatGPT Overcome Behavioral Biases in the Financial Sector? Classify-and-Rethink: Multi-Step Zero-Shot Reasoning in the Gold Investment
  3. 2023GPT-3 Models are Few-Shot Financial Reasoners
  4. 2024FinBen: A Holistic Financial Benchmark for Large Language Models
  5. 2025Reasoning or Overthinking: Evaluating Large Language Models on Financial Sentiment Analysis
  6. 2023Zero-Shot Question Answering over Financial Documents using Large Language Models
  7. 2025Large Language Models Acing Chartered Accountancy
  8. 2025FinEval-KR: A Financial Domain Evaluation Framework for Large Language Models' Knowledge and Reasoning