# Examining the Potential and Pitfalls of ChatGPT in Science and Engineering Problem-Solving

arXiv: https://arxiv.org/abs/2310.08773 · HTML: https://mlsanity.com/paper/2310.08773/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- Builds on 9 of its 31 references that are in the map.
- First posted on arXiv 12 Oct 2023.

## Builds on (most cited first, 9 of 9)

- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Large Language Models are Zero-Shot Reasoners](https://mlsanity.com/paper/2205.11916/) (2022)
- [Sparks of Artificial General Intelligence: Early experiments with GPT-4](https://mlsanity.com/paper/2303.12712/) (2023)
- [Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processing](https://mlsanity.com/paper/2107.13586/) (2021)
- [Capabilities of GPT-4 on Medical Challenge Problems](https://mlsanity.com/paper/2303.13375/) (2023)
- [How Language Model Hallucinations Can Snowball](https://mlsanity.com/paper/2305.13534/) (2023)
- [Hypothesis Search: Inductive Reasoning with Language Models](https://mlsanity.com/paper/2309.05660/) (2023)
- [LLMs and the Abstraction and Reasoning Corpus: Successes, Failures, and the Importance of Object-based Representations](https://mlsanity.com/paper/2305.18354/) (2023)
- [Why Does ChatGPT Fall Short in Providing Truthful Answers?](https://mlsanity.com/paper/2304.10513/) (2023)

## Built on by (most cited first, 11 of 11)

- [Quantum Many-Body Physics Calculations with Large Language Models](https://mlsanity.com/paper/2403.03154/) (2024)
- [Could ChatGPT get an Engineering Degree? Evaluating Higher Education Vulnerability to AI Assistants](https://mlsanity.com/paper/2408.11841/) (2024)
- [MathFlow: Enhancing the Perceptual Flow of MLLMs for Visual Mathematical Problems](https://mlsanity.com/paper/2503.16549/) (2025)
- [Evaluating GPT- and Reasoning-based Large Language Models on Physics Olympiad Problems: Surpassing Human Performance and Implications for Educational Assessment](https://mlsanity.com/paper/2505.09438/) (2025)
- [Applying IRT to Distinguish Between Human and Generative AI Responses to Multiple-Choice Assessments](https://mlsanity.com/paper/2412.02713/) (2024)
- [Assessment Design in the AI Era: A Method for Identifying Items Functioning Differentially for Humans and Chatbots](https://mlsanity.com/paper/2603.23682/) (2026)
- [Do Assessment Instruments Measure the Same Thing for Humans and LLMs? A Latent Structure Analysis](https://mlsanity.com/paper/2608.15630/) (2026)
- [Addressing accuracy and hallucination of LLMs in Alzheimer's disease research through knowledge graphs](https://mlsanity.com/paper/2508.21238/) (2025)
- [Assessing AI in Introductory Physics Problem Solving](https://mlsanity.com/paper/2607.14303/) (2026)
- [Positioning Generative Artificial Intelligence in STEM Assessment: When to Require, Scaffold, or Restrict Its Use](https://mlsanity.com/paper/2608.07475/) (2026)
- [Interpretable Humans, Alien LLMs: Expert Analysis of Latent Structures in Assessment Responses](https://mlsanity.com/paper/2608.17810/) (2026)

## Abstract (arXiv)

The study explores the capabilities of OpenAI's ChatGPT in solving different types of physics problems. ChatGPT (with GPT-4) was queried to solve a total of 40 problems from a college-level engineering physics course. These problems ranged from well-specified problems, where all data required for solving the problem was provided, to under-specified, real-world problems where not all necessary data were given. Our findings show that ChatGPT could successfully solve 62.5% of the well-specified problems, but its accuracy drops to 8.3% for under-specified problems. Analysis of the model's incorrect solutions revealed three distinct failure modes: 1) failure to construct accurate models of the physical world, 2) failure to make reasonable assumptions about missing data, and 3) calculation errors. The study offers implications for how to leverage LLM-augmented instructional materials to enhance STEM education. The insights also contribute to the broader discourse on AI's strengths and limitations, serving both educators aiming to leverage the technology and researchers investigating human-AI collaboration frameworks for problem-solving and decision-making.