# Professional Certification Benchmark Dataset: The First 500 Jobs For Large Language Models

arXiv: https://arxiv.org/abs/2305.05377 · HTML: https://mlsanity.com/paper/2305.05377/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 26 of its 53 references that are in the map.
- First posted on arXiv 7 May 2023.

## Builds on (most cited first, 26 of 26)

- [Sparks of Artificial General Intelligence: Early experiments with GPT-4](https://mlsanity.com/paper/2303.12712/) (2023)
- [A Multitask, Multilingual, Multimodal Evaluation of ChatGPT on Reasoning, Hallucination, and Interactivity](https://mlsanity.com/paper/2302.04023/) (2023)
- [AGIEval: A Human-Centric Benchmark for Evaluating Foundation Models](https://mlsanity.com/paper/2304.06364/) (2023)
- [Capabilities of GPT-4 on Medical Challenge Problems](https://mlsanity.com/paper/2303.13375/) (2023)
- [API-Bank: A Comprehensive Benchmark for Tool-Augmented LLMs](https://mlsanity.com/paper/2304.08244/) (2023)
- [Is ChatGPT a General-Purpose Natural Language Processing Task Solver?](https://mlsanity.com/paper/2302.06476/) (2023)
- [Mathematical Capabilities of ChatGPT](https://mlsanity.com/paper/2301.13867/) (2023)
- [ChatGPT: Jack of all trades, master of none](https://mlsanity.com/paper/2302.10724/) (2023)
- [Evaluating the Logical Reasoning Ability of ChatGPT and GPT-4](https://mlsanity.com/paper/2304.03439/) (2023)
- [Is ChatGPT A Good Translator? Yes With GPT-4 As The Engine](https://mlsanity.com/paper/2301.08745/) (2023)
- [A Categorical Archive of ChatGPT Failures](https://mlsanity.com/paper/2302.03494/) (2023)
- [Towards Making the Most of ChatGPT for Machine Translation](https://mlsanity.com/paper/2303.13780/) (2023)
- [How well do Large Language Models perform in Arithmetic tasks?](https://mlsanity.com/paper/2304.02015/) (2023)
- [Speak, Memory: An Archaeology of Books Known to ChatGPT/GPT-4](https://mlsanity.com/paper/2305.00118/) (2023)
- [PsyQA: A Chinese Dataset for Generating Long Counseling Text for Mental Health Support](https://mlsanity.com/paper/2106.01702/) (2021)
- [A Complete Survey on Generative AI (AIGC): Is ChatGPT from GPT-4 to GPT-5 All You Need?](https://mlsanity.com/paper/2303.11717/) (2023)
- [GPT Takes the Bar Exam](https://mlsanity.com/paper/2212.14402/) (2022)
- [Evaluating Large Language Models on a Highly-specialized Topic, Radiation Oncology Physics](https://mlsanity.com/paper/2304.01938/) (2023)
- [ParroT: Translating during Chat using Large Language Models tuned with Human Translation and Feedback](https://mlsanity.com/paper/2304.02426/) (2023)
- [Mind meets machine: Unravelling GPT-4's cognitive psychology](https://mlsanity.com/paper/2303.11436/) (2023)
- [Evaluating GPT-3.5 and GPT-4 Models on Brazilian University Admission Exams](https://mlsanity.com/paper/2303.17003/) (2023)
- [Performance of ChatGPT on the US Fundamentals of Engineering Exam: Comprehensive Assessment of Proficiency and Potential Implications for Professional Environmental Engineering Practice](https://mlsanity.com/paper/2304.12198/) (2023)
- [The Turing Deception](https://mlsanity.com/paper/2212.06721/) (2022)
- [Towards Zero-Shot and Few-Shot Table Question Answering using GPT-3](https://mlsanity.com/paper/2210.17284/) (2022)
- [Local Translation Services for Neglected Languages](https://mlsanity.com/paper/2101.01628/) (2021)
- [Creating Large Language Model Resistant Exams: Guidelines and Strategies](https://mlsanity.com/paper/2304.12203/) (2023)

## Built on by (most cited first, 2 of 2)

- [Is GPT-4 a Good Data Analyst?](https://mlsanity.com/paper/2305.15038/) (2023)
- [Visual AI and Linguistic Intelligence Through Steerability and Composability](https://mlsanity.com/paper/2312.12383/) (2023)

## Abstract (arXiv)

The research creates a professional certification survey to test large language models and evaluate their employable skills. It compares the performance of two AI models, GPT-3 and Turbo-GPT3.5, on a benchmark dataset of 1149 professional certifications, emphasizing vocational readiness rather than academic performance. GPT-3 achieved a passing score (>70% correct) in 39% of the professional certifications without fine-tuning or exam preparation. The models demonstrated qualifications in various computer-related fields, such as cloud and virtualization, business analytics, cybersecurity, network setup and repair, and data analytics. Turbo-GPT3.5 scored 100% on the valuable Offensive Security Certified Professional (OSCP) exam. The models also displayed competence in other professional domains, including nursing, licensed counseling, pharmacy, and teaching. Turbo-GPT3.5 passed the Financial Industry Regulatory Authority (FINRA) Series 6 exam with a 70% grade without preparation. Interestingly, Turbo-GPT3.5 performed well on customer service tasks, suggesting potential applications in human augmentation for chatbots in call centers and routine advice services. The models also score well on sensory and experience-based tests such as wine sommelier, beer taster, emotional quotient, and body language reader. The OpenAI model improvement from Babbage to Turbo resulted in a median 60% better-graded performance in less than a few years. This progress suggests that focusing on the latest model's shortcomings could lead to a highly performant AI capable of mastering the most demanding professional certifications. We open-source the benchmark to expand the range of testable professional skills as the models improve or gain emergent capabilities.