# GPT as Knowledge Worker: A Zero-Shot Evaluation of (AI)CPA Capabilities

arXiv: https://arxiv.org/abs/2301.04408 · HTML: https://mlsanity.com/paper/2301.04408/

## Key facts

- Cited by 22 papers in the map (data through 14 Oct 2026).
- Builds on 16 of its 43 references that are in the map.
- First posted on arXiv 11 Jan 2023.

## Builds on (most cited first, 16 of 16)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Distributed Representations of Words and Phrases and their Compositionality](https://mlsanity.com/paper/1310.4546/) (2013)
- [Deep contextualized word representations](https://mlsanity.com/paper/1802.05365/) (2018)
- [Survey of Hallucination in Natural Language Generation](https://mlsanity.com/paper/2202.03629/) (2022)
- [BLOOM: A 176B-Parameter Open-Access Multilingual Language Model](https://mlsanity.com/paper/2211.05100/) (2022)
- [Big Bird: Transformers for Longer Sequences](https://mlsanity.com/paper/2007.14062/) (2020)
- [LaMDA: Language Models for Dialog Applications](https://mlsanity.com/paper/2201.08239/) (2022)
- [Show Your Work: Scratchpads for Intermediate Computation with Language Models](https://mlsanity.com/paper/2112.00114/) (2021)
- [Large Language Models Can Self-Improve](https://mlsanity.com/paper/2210.11610/) (2022)
- [Limitations of Language Models in Arithmetic and Symbolic Induction](https://mlsanity.com/paper/2208.05051/) (2022)
- [GPT Takes the Bar Exam](https://mlsanity.com/paper/2212.14402/) (2022)
- [Structured information extraction from complex scientific text with fine-tuned large language models](https://mlsanity.com/paper/2212.05238/) (2022)
- [Improving Language Model Prompting in Support of Semi-autonomous Task Learning](https://mlsanity.com/paper/2209.07636/) (2022)
- [Overcoming Barriers to Skill Injection in Language Modeling: Case Study in Arithmetic](https://mlsanity.com/paper/2211.02098/) (2022)

## Built on by (most cited first, 22 of 22)

- [Challenges and Applications of Large Language Models](https://mlsanity.com/paper/2307.10169/) (2023)
- [Domain Specialization as the Key to Make Large Language Models Disruptive: A Comprehensive Survey](https://mlsanity.com/paper/2305.18703/) (2023)
- [Thrilled by Your Progress! Large Language Models (GPT-4) No Longer Struggle to Pass Assessments in Higher Education Programming Courses](https://mlsanity.com/paper/2306.10073/) (2023)
- [Natural Language Processing in the Legal Domain](https://mlsanity.com/paper/2302.12039/) (2023)
- [ChatGPT may Pass the Bar Exam soon, but has a Long Way to Go for the LexGLUE benchmark](https://mlsanity.com/paper/2304.12202/) (2023)
- [Large Language Models (GPT) Struggle to Answer Multiple-Choice Questions about Code](https://mlsanity.com/paper/2303.08033/) (2023)
- [Impossible Distillation: from Low-Quality Model to High-Quality Dataset & Model for Summarization and Paraphrasing](https://mlsanity.com/paper/2305.16635/) (2023)
- [Can GPT-4 Support Analysis of Textual Data in Tasks Requiring Highly Specialized Domain Expertise?](https://mlsanity.com/paper/2306.13906/) (2023)
- [Systematic Evaluation of GPT-3 for Zero-Shot Personality Estimation](https://mlsanity.com/paper/2306.01183/) (2023)
- [AI for social science and social science of AI: A Survey](https://mlsanity.com/paper/2401.11839/) (2024)
- [Evaluating GPT-3.5 and GPT-4 Models on Brazilian University Admission Exams](https://mlsanity.com/paper/2303.17003/) (2023)
- [Equipping Language Models with Tool Use Capability for Tabular Data Analysis in Finance](https://mlsanity.com/paper/2401.15328/) (2024)
- [GPT has become financially literate: Insights from financial literacy tests of GPT and a preliminary test of how people use it as a source of advice](https://mlsanity.com/paper/2309.00649/) (2023)
- [Evaluating GPT-4's Vision Capabilities on Brazilian University Admission Exams](https://mlsanity.com/paper/2311.14169/) (2023)
- [Large Scale Foundation Models for Intelligent Manufacturing Applications: A Survey](https://mlsanity.com/paper/2312.06718/) (2023)
- [Evaluating Large Language Models for Public Health Classification and Extraction Tasks](https://mlsanity.com/paper/2405.14766/) (2024)
- [A Scoping Review of ChatGPT Research in Accounting and Finance](https://mlsanity.com/paper/2412.05731/) (2024)
- [It Cannot Be Right If It Was Written by AI: On Lawyers' Preferences of Documents Perceived as Authored by an LLM vs a Human](https://mlsanity.com/paper/2407.06798/) (2024)
- [Healthy LLMs? Benchmarking LLM Knowledge of UK Government Public Health Information](https://mlsanity.com/paper/2505.06046/) (2025)
- [Assessing the Capability of LLMs in Solving POSCOMP Questions](https://mlsanity.com/paper/2505.20338/) (2025)
- [Evaluating LLMs for Police Decision-Making: A Framework Based on Police Action Scenarios](https://mlsanity.com/paper/2601.03553/) (2026)
- [Can Generative Pre-trained Transformers (GPT) Pass Assessments in Higher Education Programming Courses?](https://mlsanity.com/paper/2303.09325/) (2023)

## Abstract (arXiv)

The global economy is increasingly dependent on knowledge workers to meet the needs of public and private organizations. While there is no single definition of knowledge work, organizations and industry groups still attempt to measure individuals' capability to engage in it. The most comprehensive assessment of capability readiness for professional knowledge workers is the Uniform CPA Examination developed by the American Institute of Certified Public Accountants (AICPA). In this paper, we experimentally evaluate OpenAI's `text-davinci-003` and prior versions of GPT on both a sample Regulation (REG) exam and an assessment of over 200 multiple-choice questions based on the AICPA Blueprints for legal, financial, accounting, technology, and ethical tasks. First, we find that `text-davinci-003` achieves a correct rate of 14.4% on a sample REG exam section, significantly underperforming human capabilities on quantitative reasoning in zero-shot prompts. Second, `text-davinci-003` appears to be approaching human-level performance on the Remembering & Understanding and Application skill levels in the Exam absent calculation. For best prompt and parameters, the model answers 57.6% of questions correctly, significantly better than the 25% guessing rate, and its top two answers are correct 82.1% of the time, indicating strong non-entailment. Finally, we find that recent generations of GPT-3 demonstrate material improvements on this assessment, rising from 30% for `text-davinci-001` to 57% for `text-davinci-003`. These findings strongly suggest that large language models have the potential to transform the quality and efficiency of future knowledge work.