# Multilingual Performance of a Multimodal Artificial Intelligence System on Multisubject Physics Concept Inventories

arXiv: https://arxiv.org/abs/2501.06143 · HTML: https://mlsanity.com/paper/2501.06143/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 13 of its 162 references that are in the map.
- First posted on arXiv 10 Jan 2025.

## Builds on (most cited first, 13 of 13)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [PHYRE: A New Benchmark for Physical Reasoning](https://mlsanity.com/paper/1908.05656/) (2019)
- [People cannot distinguish GPT-4 from a human in a Turing test](https://mlsanity.com/paper/2405.08007/) (2024)
- [Lost in Translation: Large Language Models in Non-English Content Analysis](https://mlsanity.com/paper/2306.07377/) (2023)
- [Using Generative AI and Multi-Agents to Provide Automatic Feedback](https://mlsanity.com/paper/2411.07407/) (2024)
- [Teaching LLMs to Abstain across Languages via Multilingual Feedback](https://mlsanity.com/paper/2406.15948/) (2024)
- [AI and the FCI: Can ChatGPT Project an Understanding of Introductory Physics?](https://mlsanity.com/paper/2303.01067/) (2023)
- [PhysicsAssistant: An LLM-Powered Interactive Learning Robot for Physics Lab Investigations](https://mlsanity.com/paper/2403.18721/) (2024)
- [Enhancing STEM Learning with ChatGPT and Bing Chat as Objects to Think With: A Case Study](https://mlsanity.com/paper/2305.02202/) (2023)
- [Using artificial-intelligence tools to make LaTeX content accessible to blind readers](https://mlsanity.com/paper/2306.02480/) (2023)
- [LLM-Generated Tips Rival Expert-Created Tips in Helping Students Answer Quantum-Computing Questions](https://mlsanity.com/paper/2407.17024/) (2024)
- [A Personalised Learning Tool for Physics Undergraduate Students Built On a Large Language Model for Symbolic Regression](https://mlsanity.com/paper/2407.00065/) (2024)

## Built on by (most cited first, 5 of 5)

- [Evaluating GPT- and Reasoning-based Large Language Models on Physics Olympiad Problems: Surpassing Human Performance and Implications for Educational Assessment](https://mlsanity.com/paper/2505.09438/) (2025)
- [Daily and Weekly Periodicity in Large Language Model Performance and Its Implications for Research](https://mlsanity.com/paper/2602.15889/) (2026)
- [LLM-as-a-judge validity is strongly task-dependent across physics assessment formats](https://mlsanity.com/paper/2603.14732/) (2026)
- [Feedback Indices to Evaluate LLM Responses to Rebuttals for Multiple Choice Type Questions](https://mlsanity.com/paper/2601.03285/) (2026)
- [Assessing AI in Introductory Physics Problem Solving](https://mlsanity.com/paper/2607.14303/) (2026)

## Abstract (arXiv)

We investigate the multilingual and multimodal performance of a large language model-based artificial intelligence (AI) system, GPT-4o, using a diverse set of physics concept inventories spanning multiple languages and subject categories. The inventories, sourced from the PhysPort website, cover classical physics topics such as mechanics, electromagnetism, optics, and thermodynamics, as well as relativity, quantum mechanics, astronomy, mathematics, and laboratory skills. Unlike previous text-only studies, we uploaded the inventories as images to reflect what a student would see on paper, thereby assessing the system's multimodal functionality. Our results indicate variation in performance across subjects, with laboratory skills standing out as the weakest. We also observe differences across languages, with English and European languages showing the strongest performance. Notably, the relative difficulty of an inventory item is largely independent of the language of the survey. When comparing AI results to existing literature on student performance, we find that the AI system outperforms average post-instruction undergraduate students in all subject categories except laboratory skills. Furthermore, the AI performs worse on items requiring visual interpretation of images than on those that are purely text-based. While our exploratory findings show GPT-4o's potential usefulness in physics education, they highlight the critical need for instructors to foster students' ability to critically evaluate AI outputs, adapt curricula thoughtfully in response to AI advancements, and address equity concerns associated with AI integration.