# Advances in apparent conceptual physics reasoning in GPT-4

arXiv: https://arxiv.org/abs/2303.17012 · HTML: https://mlsanity.com/paper/2303.17012/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- Placed in Spatial and embodied reasoning, one of 17 directions in LLM reasoning.
- Builds on 7 of its 35 references that are in the map.
- First posted on arXiv 29 Mar 2023.

## Placed in

- LLM reasoning / [Spatial and embodied reasoning](https://mlsanity.com/lr/spatial-and-embodied-reasoning/)

## Builds on (most cited first, 7 of 7)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Sparks of Artificial General Intelligence: Early experiments with GPT-4](https://mlsanity.com/paper/2303.12712/) (2023)
- [GPT Takes the Bar Exam](https://mlsanity.com/paper/2212.14402/) (2022)
- [AI and the FCI: Can ChatGPT Project an Understanding of Introductory Physics?](https://mlsanity.com/paper/2303.01067/) (2023)

## Built on by (most cited first, 12 of 12)

- [Fundamental Limitations of Alignment in Large Language Models](https://mlsanity.com/paper/2304.11082/) (2023)
- [Towards Robust and Secure Embodied AI: A Survey on Vulnerabilities and Attacks](https://mlsanity.com/paper/2502.13175/) (2025)
- [Tradeoffs Between Alignment and Helpfulness in Language Models with Steering Methods](https://mlsanity.com/paper/2401.16332/) (2024)
- [When does compositional structure yield compositional generalization? A kernel theory](https://mlsanity.com/paper/2405.16391/) (2024)
- [Unreflected Acceptance -- Investigating the Negative Consequences of ChatGPT-Assisted Problem Solving in Physics Education](https://mlsanity.com/paper/2309.03087/) (2023)
- [Testing LLM performance on the Physics GRE: some observations](https://mlsanity.com/paper/2312.04613/) (2023)
- [Compositional Hardness of Code in Large Language Models -- A Probabilistic Perspective](https://mlsanity.com/paper/2409.18028/) (2024)
- [Should We Fear Large Language Models? A Structural Analysis of the Human Reasoning System for Elucidating LLM Capabilities and Risks Through the Lens of Heidegger's Philosophy](https://mlsanity.com/paper/2403.03288/) (2024)
- [A criterion for Artificial General Intelligence: hypothetic-deductive reasoning, tested on ChatGPT](https://mlsanity.com/paper/2308.02950/) (2023)
- [Scaffold or Crutch? Examining College Students' Use and Views of Generative AI Tools for STEM Education](https://mlsanity.com/paper/2412.02653/) (2024)
- [Causes in neuron diagrams, and testing causal reasoning in Large Language Models. A glimpse of the future of philosophy?](https://mlsanity.com/paper/2506.14239/) (2025)
- [Feedback Indices to Evaluate LLM Responses to Rebuttals for Multiple Choice Type Questions](https://mlsanity.com/paper/2601.03285/) (2026)

## Abstract (arXiv)

ChatGPT is built on a large language model trained on an enormous corpus of human text to emulate human conversation. Despite lacking any explicit programming regarding the laws of physics, recent work has demonstrated that GPT-3.5 could pass an introductory physics course at some nominal level and register something close to a minimal understanding of Newtonian Mechanics on the Force Concept Inventory. This work replicates those results and also demonstrates that the latest version, GPT-4, has reached a much higher mark in the latter context. Indeed, its responses come quite close to perfectly demonstrating expert-level competence, with a few very notable exceptions and limitations. We briefly comment on the implications of this for the future of physics education and pedagogy.