# Assessing Large Language Models in Mechanical Engineering Education: A Study on Mechanics-Focused Conceptual Understanding arXiv: https://arxiv.org/abs/2401.12983 ยท HTML: https://mlsanity.com/paper/2401.12983/ ## Key facts - Cited by 15 papers in the map (data through 14 Oct 2026). - Builds on 17 of its 41 references that are in the map. - First posted on arXiv 13 Jan 2024. ## Builds on (most cited first, 17 of 17) - [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023) - [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022) - [GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding](https://mlsanity.com/paper/1804.07461/) (2018) - [BioBERT: a pre-trained biomedical language representation model for biomedical text mining](https://mlsanity.com/paper/1901.08746/) (2019) - [SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systems](https://mlsanity.com/paper/1905.00537/) (2019) - [The LAMBADA dataset: Word prediction requiring a broad discourse context](https://mlsanity.com/paper/1606.06031/) (2016) - [Is ChatGPT a General-Purpose Natural Language Processing Task Solver?](https://mlsanity.com/paper/2302.06476/) (2023) - [Augmented Language Models: a Survey](https://mlsanity.com/paper/2302.07842/) (2023) - [14 Examples of How LLMs Can Transform Materials Science and Chemistry: A Reflection on a Large Language Model Hackathon](https://mlsanity.com/paper/2306.06283/) (2023) - [When Brain-inspired AI Meets AGI](https://mlsanity.com/paper/2303.15935/) (2023) - [Evaluating Large Language Models on a Highly-specialized Topic, Radiation Oncology Physics](https://mlsanity.com/paper/2304.01938/) (2023) - [MechGPT, a language-based strategy for mechanics and materials modeling that connects knowledge across scales, disciplines and modalities](https://mlsanity.com/paper/2310.10445/) (2023) - [Large Language Models as Master Key: Unlocking the Secrets of Materials Science with GPT](https://mlsanity.com/paper/2304.02213/) (2023) - [BioinspiredLLM: Conversational Large Language Model for the Mechanics of Biological and Bio-inspired Materials](https://mlsanity.com/paper/2309.08788/) (2023) - [MeLM, a generative pretrained language modeling framework that solves forward and inverse mechanics problems](https://mlsanity.com/paper/2306.17525/) (2023) - [MaScQA: A Question Answering Dataset for Investigating Materials Science Knowledge of Large Language Models](https://mlsanity.com/paper/2308.09115/) (2023) - [Evaluating the Potential of Leading Large Language Models in Reasoning Biology Questions](https://mlsanity.com/paper/2311.07582/) (2023) ## Built on by (most cited first, 15 of 15) - [A Review of Large Language Models and Autonomous Agents in Chemistry](https://mlsanity.com/paper/2407.01603/) (2024) - [Opportunities and Challenges of Large Language Models for Low-Resource Languages in Humanities Research](https://mlsanity.com/paper/2412.04497/) (2024) - [Large Language Models for Manufacturing](https://mlsanity.com/paper/2410.21418/) (2024) - [A Systematic Assessment of OpenAI o1-Preview for Higher Order Thinking in Education](https://mlsanity.com/paper/2410.21287/) (2024) - [Transcending Language Boundaries: Harnessing LLMs for Low-Resource Language Translation](https://mlsanity.com/paper/2411.11295/) (2024) - [A Systematic Review on Prompt Engineering in Large Language Models for K-12 STEM Education](https://mlsanity.com/paper/2410.11123/) (2024) - [Foundation Models for Low-Resource Language Education (Vision Paper)](https://mlsanity.com/paper/2412.04774/) (2024) - [Legal Evalutions and Challenges of Large Language Models](https://mlsanity.com/paper/2411.10137/) (2024) - [Large Language Models for Extrapolative Modeling of Manufacturing Processes](https://mlsanity.com/paper/2502.12185/) (2025) - [Large Language Models for Bioinformatics](https://mlsanity.com/paper/2501.06271/) (2025) - [The Lazy Student's Dream: ChatGPT Passing an Engineering Course on Its Own](https://mlsanity.com/paper/2503.05760/) (2025) - [Human-in-the-Loop: Quantitative Evaluation of 3D Models Generation by Large Language Models](https://mlsanity.com/paper/2509.07010/) (2025) - [CAMB: A comprehensive industrial LLM benchmark on civil aviation maintenance](https://mlsanity.com/paper/2508.20420/) (2025) - [Large Language Models and their Awareness of Mechanics and Spatial Geometry](https://mlsanity.com/paper/2608.14615/) (2026) - [A Comparative Study of Recent Large Language Models on Generating Hospital Discharge Summaries for Lung Cancer Patients](https://mlsanity.com/paper/2411.03805/) (2024) ## Abstract (arXiv) This study is a pioneering endeavor to investigate the capabilities of Large Language Models (LLMs) in addressing conceptual questions within the domain of mechanical engineering with a focus on mechanics. Our examination involves a manually crafted exam encompassing 126 multiple-choice questions, spanning various aspects of mechanics courses, including Fluid Mechanics, Mechanical Vibration, Engineering Statics and Dynamics, Mechanics of Materials, Theory of Elasticity, and Continuum Mechanics. Three LLMs, including ChatGPT (GPT-3.5), ChatGPT (GPT-4), and Claude (Claude-2.1), were subjected to evaluation against engineering faculties and students with or without mechanical engineering background. The findings reveal GPT-4's superior performance over the other two LLMs and human cohorts in answering questions across various mechanics topics, except for Continuum Mechanics. This signals the potential future improvements for GPT models in handling symbolic calculations and tensor analyses. The performances of LLMs were all significantly improved with explanations prompted prior to direct responses, underscoring the crucial role of prompt engineering. Interestingly, GPT-3.5 demonstrates improved performance with prompts covering a broader domain, while GPT-4 excels with prompts focusing on specific subjects. Finally, GPT-4 exhibits notable advancements in mitigating input bias, as evidenced by guessing preferences for humans. This study unveils the substantial potential of LLMs as highly knowledgeable assistants in both mechanical pedagogy and scientific research.