mlsanityData to 14 Oct 2026

Examining the Potential and Pitfalls of ChatGPT in Science and Engineering Problem-Solving

arXiv 2310.08773 ↗First posted 12 Oct 2023cs.AI cs.CEcited by 11

Not on the map. Why?

Abstract (arXiv)

The study explores the capabilities of OpenAI's ChatGPT in solving different types of physics problems. ChatGPT (with GPT-4) was queried to solve a total of 40 problems from a college-level engineering physics course. These problems ranged from well-specified problems, where all data required for solving the problem was provided, to under-specified, real-world problems where not all necessary data were given. Our findings show that ChatGPT could successfully solve 62.5% of the well-specified problems, but its accuracy drops to 8.3% for under-specified problems. Analysis of the model's incorrect solutions revealed three distinct failure modes: 1) failure to construct accurate models of the physical world, 2) failure to make reasonable assumptions about missing data, and 3) calculation errors. The study offers implications for how to leverage LLM-augmented instructional materials to enhance STEM education. The insights also contribute to the broader discourse on AI's strengths and limitations, serving both educators aiming to leverage the technology and researchers investigating human-AI collaboration frameworks for problem-solving and decision-making.

Key facts. Cited by 11 papers in the map (data through 14 Oct 2026). Builds on 9 of its 31 references that are in the map. First posted on arXiv 12 Oct 2023.

Builds on

9 of 31 references.

References · 9

  1. 2023GPT-4 Technical Reportcited by 18,396
  2. 2022Large Language Models are Zero-Shot Reasonerscited by 4,374
  3. 2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,592
  4. 2021Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processingcited by 2,234
  5. 2023Capabilities of GPT-4 on Medical Challenge Problemscited by 638
  6. 2023How Language Model Hallucinations Can Snowballcited by 311
  7. 2023Hypothesis Search: Inductive Reasoning with Language Modelscited by 120
  8. 2023LLMs and the Abstraction and Reasoning Corpus: Successes, Failures, and the Importance of Object-based Representationscited by 82
  9. 2023Why Does ChatGPT Fall Short in Providing Truthful Answers?cited by 54

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 3; 2025: 3; 2026: 5.

11 papers.

  1. 2024Quantum Many-Body Physics Calculations with Large Language Modelscited by 17
  2. 2024Could ChatGPT get an Engineering Degree? Evaluating Higher Education Vulnerability to AI Assistantscited by 10
  3. 2025MathFlow: Enhancing the Perceptual Flow of MLLMs for Visual Mathematical Problemscited by 10
  4. 2025Evaluating GPT- and Reasoning-based Large Language Models on Physics Olympiad Problems: Surpassing Human Performance and Implications for Educational Assessmentcited by 7
  5. 2024Applying IRT to Distinguish Between Human and Generative AI Responses to Multiple-Choice Assessmentscited by 3
  6. 2026Assessment Design in the AI Era: A Method for Identifying Items Functioning Differentially for Humans and Chatbotscited by 1
  7. 2026Do Assessment Instruments Measure the Same Thing for Humans and LLMs? A Latent Structure Analysiscited by 1
  8. 2025Addressing accuracy and hallucination of LLMs in Alzheimer's disease research through knowledge graphscited by 0
  9. 2026Assessing AI in Introductory Physics Problem Solvingcited by 0
  10. 2026Positioning Generative Artificial Intelligence in STEM Assessment: When to Require, Scaffold, or Restrict Its Usecited by 0
  11. 2026Interpretable Humans, Alien LLMs: Expert Analysis of Latent Structures in Assessment Responsescited by 0