# AI and the FCI: Can ChatGPT Project an Understanding of Introductory Physics?

arXiv: https://arxiv.org/abs/2303.01067 · HTML: https://mlsanity.com/paper/2303.01067/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- Builds on 5 of its 52 references that are in the map.
- First posted on arXiv 2 Mar 2023.

## Builds on (most cited first, 5 of 5)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Deep reinforcement learning from human preferences](https://mlsanity.com/paper/1706.03741/) (2017)
- [GPT Takes the Bar Exam](https://mlsanity.com/paper/2212.14402/) (2022)

## Built on by (most cited first, 13 of 13)

- [Summary of ChatGPT-Related Research and Perspective Towards the Future of Large Language Models](https://mlsanity.com/paper/2304.01852/) (2023)
- [ChatGPT vs Human-authored Text: Insights into Controllable Text Summarization and Sentence Style Transfer](https://mlsanity.com/paper/2306.07799/) (2023)
- [Augmented Large Language Models with Parametric Knowledge Guiding](https://mlsanity.com/paper/2305.04757/) (2023)
- [Humanizing Machine-Generated Content: Evading AI-Text Detection through Adversarial Attack](https://mlsanity.com/paper/2404.01907/) (2024)
- [Advances in apparent conceptual physics reasoning in GPT-4](https://mlsanity.com/paper/2303.17012/) (2023)
- [A comparison of Human, GPT-3.5, and GPT-4 Performance in a University-Level Coding Course](https://mlsanity.com/paper/2403.16977/) (2024)
- [Towards a Benchmark for Scientific Understanding in Humans and Machines](https://mlsanity.com/paper/2304.10327/) (2023)
- [TH-Bench: Evaluating Evading Attacks via Humanizing AI Text on Machine-Generated Text Detectors](https://mlsanity.com/paper/2503.08708/) (2025)
- [MedGPTEval: A Dataset and Benchmark to Evaluate Responses of Large Language Models in Medicine](https://mlsanity.com/paper/2305.07340/) (2023)
- [Unreflected Acceptance -- Investigating the Negative Consequences of ChatGPT-Assisted Problem Solving in Physics Education](https://mlsanity.com/paper/2309.03087/) (2023)
- [Evaluating GPT- and Reasoning-based Large Language Models on Physics Olympiad Problems: Surpassing Human Performance and Implications for Educational Assessment](https://mlsanity.com/paper/2505.09438/) (2025)
- [Multilingual Performance of a Multimodal Artificial Intelligence System on Multisubject Physics Concept Inventories](https://mlsanity.com/paper/2501.06143/) (2025)
- [Enhancing Spectral Knowledge Interrogation: A Reliable Retrieval-Augmented Generative Framework on Large Language Models](https://mlsanity.com/paper/2408.11557/) (2024)

## Abstract (arXiv)

ChatGPT is a groundbreaking ``chatbot"--an AI interface built on a large language model that was trained on an enormous corpus of human text to emulate human conversation. Beyond its ability to converse in a plausible way, it has attracted attention for its ability to competently answer questions from the bar exam and from MBA coursework, and to provide useful assistance in writing computer code. These apparent abilities have prompted discussion of ChatGPT as both a threat to the integrity of higher education and conversely as a powerful teaching tool. In this work we present a preliminary analysis of how two versions of ChatGPT (ChatGPT3.5 and ChatGPT4) fare in the field of first-semester university physics, using a modified version of the Force Concept Inventory (FCI) to assess whether it can give correct responses to conceptual physics questions about kinematics and Newtonian dynamics. We demonstrate that, by some measures, ChatGPT3.5 can match or exceed the median performance of a university student who has completed one semester of college physics, though its performance is notably uneven and the results are nuanced. By these same measures, we find that ChatGPT4's performance is approaching the point of being indistinguishable from that of an expert physicist when it comes to introductory mechanics topics. After the completion of our work we became aware of Ref [1], which preceded us to publication and which completes an extensive analysis of the abilities of ChatGPT3.5 in a physics class, including a different modified version of the FCI. We view this work as confirming that portion of their results, and extending the analysis to ChatGPT4, which shows rapid and notable improvement in most, but not all respects.