# ChatGPT & Mechanical Engineering: Examining performance on the FE Mechanical Engineering and Undergraduate Exams

arXiv: https://arxiv.org/abs/2309.15866 · HTML: https://mlsanity.com/paper/2309.15866/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 16 of its 44 references that are in the map.
- First posted on arXiv 26 Sep 2023.

## Builds on (most cited first, 16 of 16)

- [How is ChatGPT's behavior changing over time?](https://mlsanity.com/paper/2307.09009/) (2023)
- [Mathematical Capabilities of ChatGPT](https://mlsanity.com/paper/2301.13867/) (2023)
- [A Categorical Archive of ChatGPT Failures](https://mlsanity.com/paper/2302.03494/) (2023)
- [How well do Large Language Models perform in Arithmetic tasks?](https://mlsanity.com/paper/2304.02015/) (2023)
- [A Systematic Study and Comprehensive Evaluation of ChatGPT on Benchmark Datasets](https://mlsanity.com/paper/2305.18486/) (2023)
- [Evaluating GPT-4 and ChatGPT on Japanese Medical Licensing Examinations](https://mlsanity.com/paper/2303.18027/) (2023)
- [An Independent Evaluation of ChatGPT on Mathematical Word Problems (MWP)](https://mlsanity.com/paper/2302.13814/) (2023)
- [ChatGPT Participates in a Computer Science Exam](https://mlsanity.com/paper/2303.09461/) (2023)
- [ChatGPT is a Remarkable Tool -- For Experts](https://mlsanity.com/paper/2306.03102/) (2023)
- [Performance of ChatGPT on the US Fundamentals of Engineering Exam: Comprehensive Assessment of Proficiency and Potential Implications for Professional Environmental Engineering Practice](https://mlsanity.com/paper/2304.12198/) (2023)
- [Can ChatGPT Pass An Introductory Level Functional Language Programming Course?](https://mlsanity.com/paper/2305.02230/) (2023)
- [Industrial Engineering with Large Language Models: A case study of ChatGPT's performance on Oil & Gas problems](https://mlsanity.com/paper/2304.14354/) (2023)
- [Analyzing ChatGPT's Aptitude in an Introductory Computer Engineering Course](https://mlsanity.com/paper/2304.06122/) (2023)
- [Humans are Still Better than ChatGPT: Case of the IEEEXtreme Competition](https://mlsanity.com/paper/2305.06934/) (2023)
- [Chatbots put to the test in math and logic problems: A preliminary comparison and assessment of ChatGPT-3.5, ChatGPT-4, and Google Bard](https://mlsanity.com/paper/2305.18618/) (2023)
- [Cognitive network science reveals bias in GPT-3, ChatGPT, and GPT-4 mirroring math anxiety in high-school students](https://mlsanity.com/paper/2305.18320/) (2023)

## Abstract (arXiv)

The launch of ChatGPT at the end of 2022 generated large interest into possible applications of artificial intelligence in STEM education and among STEM professions. As a result many questions surrounding the capabilities of generative AI tools inside and outside of the classroom have been raised and are starting to be explored. This study examines the capabilities of ChatGPT within the discipline of mechanical engineering. It aims to examine use cases and pitfalls of such a technology in the classroom and professional settings. ChatGPT was presented with a set of questions from junior and senior level mechanical engineering exams provided at a large private university, as well as a set of practice questions for the Fundamentals of Engineering Exam (FE) in Mechanical Engineering. The responses of two ChatGPT models, one free to use and one paid subscription, were analyzed. The paper found that the subscription model (GPT-4) greatly outperformed the free version (GPT-3.5), achieving 76% correct vs 51% correct, but the limitation of text only input on both models makes neither likely to pass the FE exam. The results confirm findings in the literature with regards to types of errors and pitfalls made by ChatGPT. It was found that due to its inconsistency and a tendency to confidently produce incorrect answers the tool is best suited for users with expert knowledge.