# Can GPT models be Financial Analysts? An Evaluation of ChatGPT and GPT-4 on mock CFA Exams

arXiv: https://arxiv.org/abs/2310.08678 · HTML: https://mlsanity.com/paper/2310.08678/

## Key facts

- Cited by 18 papers in the map (data through 14 Oct 2026).
- Builds on 21 of its 53 references that are in the map.
- First posted on arXiv 12 Oct 2023.

## Builds on (most cited first, 21 of 21)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks](https://mlsanity.com/paper/2005.11401/) (2020)
- [Large Language Models are Zero-Shot Reasoners](https://mlsanity.com/paper/2205.11916/) (2022)
- [BloombergGPT: A Large Language Model for Finance](https://mlsanity.com/paper/2303.17564/) (2023)
- [A Multitask, Multilingual, Multimodal Evaluation of ChatGPT on Reasoning, Hallucination, and Interactivity](https://mlsanity.com/paper/2302.04023/) (2023)
- [UnifiedQA: Crossing Format Boundaries With a Single QA System](https://mlsanity.com/paper/2005.00700/) (2020)
- [OCR-free Document Understanding Transformer](https://mlsanity.com/paper/2111.15664/) (2021)
- [FinBERT: Financial Sentiment Analysis with Pre-trained Language Models](https://mlsanity.com/paper/1908.10063/) (2019)
- [CodeRL: Mastering Code Generation through Pretrained Models and Deep Reinforcement Learning](https://mlsanity.com/paper/2207.01780/) (2022)
- [Causal Reasoning and Large Language Models: Opening a New Frontier for Causality](https://mlsanity.com/paper/2305.00050/) (2023)
- [Mathematical Capabilities of ChatGPT](https://mlsanity.com/paper/2301.13867/) (2023)
- [SciBench: Evaluating College-Level Scientific Problem-Solving Abilities of Large Language Models](https://mlsanity.com/paper/2307.10635/) (2023)
- [ConvFinQA: Exploring the Chain of Numerical Reasoning in Conversational Finance Question Answering](https://mlsanity.com/paper/2210.03849/) (2022)
- [Few-shot Fine-tuning vs. In-context Learning: A Fair Comparison and Evaluation](https://mlsanity.com/paper/2305.16938/) (2023)
- [Unified Named Entity Recognition as Word-Word Relation Classification](https://mlsanity.com/paper/2112.10070/) (2021)
- [Extractive Summarization via ChatGPT for Faithful Summary Generation](https://mlsanity.com/paper/2304.04193/) (2023)
- [A Numerical Reasoning Question Answering System with Fine-grained Retriever and the Ensemble of Multiple Generators for FinQA](https://mlsanity.com/paper/2206.08506/) (2022)
- [Exploring the Effectiveness of GPT Models in Test-Taking: A Case Study of the Driver's License Knowledge Test](https://mlsanity.com/paper/2308.11827/) (2023)

## Built on by (most cited first, 18 of 18)

- [FinanceBench: A New Benchmark for Financial Question Answering](https://mlsanity.com/paper/2311.11944/) (2023)
- [DocMath-Eval: Evaluating Math Reasoning Capabilities of LLMs in Understanding Long and Specialized Documents](https://mlsanity.com/paper/2311.09805/) (2023)
- [Can Large Language Models Beat Wall Street? Unveiling the Potential of AI in Stock Selection](https://mlsanity.com/paper/2401.03737/) (2024)
- [Demystifying Domain-adaptive Post-training for Financial LLMs](https://mlsanity.com/paper/2501.04961/) (2025)
- [FinLoRA: Benchmarking LoRA Methods for Fine-Tuning LLMs on Financial Datasets](https://mlsanity.com/paper/2505.19819/) (2025)
- [Evaluating Large Language Models as Expert Annotators](https://mlsanity.com/paper/2508.07827/) (2025)
- [EconNLI: Evaluating Large Language Models on Economics Reasoning](https://mlsanity.com/paper/2407.01212/) (2024)
- [Adversarial Math Word Problem Generation](https://mlsanity.com/paper/2402.17916/) (2024)
- [Advanced Financial Reasoning at Scale: A Comprehensive Evaluation of Large Language Models on CFA Level III](https://mlsanity.com/paper/2507.02954/) (2025)
- [FinTexTS: Financial Text-Paired Time-Series Dataset via Semantic-Based and Multi-Level Pairing](https://mlsanity.com/paper/2603.02702/) (2026)
- [Financial Knowledge Large Language Model](https://mlsanity.com/paper/2407.00365/) (2024)
- [Are Expert-Level Language Models Expert-Level Annotators?](https://mlsanity.com/paper/2410.03254/) (2024)
- [Quantifying Qualitative Insights: Leveraging LLMs to Market Predict](https://mlsanity.com/paper/2411.08404/) (2024)
- [Evaluating Large Language Models for Financial Reasoning: A CFA-Based Benchmark Study](https://mlsanity.com/paper/2509.04468/) (2025)
- [THaLLE: Text Hyperlocally Augmented Large Language Extension -- Technical Report](https://mlsanity.com/paper/2406.07505/) (2024)
- [AI Analyst: Framework and Comprehensive Evaluation of Large Language Models for Financial Time Series Report Generation](https://mlsanity.com/paper/2507.00718/) (2025)
- [KAHAN: Knowledge-Augmented Hierarchical Analysis and Narration for Financial Data Narration](https://mlsanity.com/paper/2509.17037/) (2025)
- [Reasoning Models Ace the CFA Exams](https://mlsanity.com/paper/2512.08270/) (2025)

## Abstract (arXiv)

Large Language Models (LLMs) have demonstrated remarkable performance on a wide range of Natural Language Processing (NLP) tasks, often matching or even beating state-of-the-art task-specific models. This study aims at assessing the financial reasoning capabilities of LLMs. We leverage mock exam questions of the Chartered Financial Analyst (CFA) Program to conduct a comprehensive evaluation of ChatGPT and GPT-4 in financial analysis, considering Zero-Shot (ZS), Chain-of-Thought (CoT), and Few-Shot (FS) scenarios. We present an in-depth analysis of the models' performance and limitations, and estimate whether they would have a chance at passing the CFA exams. Finally, we outline insights into potential strategies and improvements to enhance the applicability of LLMs in finance. In this perspective, we hope this work paves the way for future studies to continue enhancing LLMs for financial reasoning through rigorous evaluation.