# Testing LLM performance on the Physics GRE: some observations

arXiv: https://arxiv.org/abs/2312.04613 · HTML: https://mlsanity.com/paper/2312.04613/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 11 of its 20 references that are in the map.
- First posted on arXiv 7 Dec 2023.

## Builds on (most cited first, 11 of 11)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Measuring Massive Multitask Language Understanding](https://mlsanity.com/paper/2009.03300/) (2020)
- [GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding](https://mlsanity.com/paper/1804.07461/) (2018)
- [Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge](https://mlsanity.com/paper/1803.05457/) (2018)
- [QLoRA: Efficient Finetuning of Quantized LLMs](https://mlsanity.com/paper/2305.14314/) (2023)
- [BLOOM: A 176B-Parameter Open-Access Multilingual Language Model](https://mlsanity.com/paper/2211.05100/) (2022)
- [LaMDA: Language Models for Dialog Applications](https://mlsanity.com/paper/2201.08239/) (2022)
- [Ludwig: a type-based declarative deep learning toolbox](https://mlsanity.com/paper/1909.07930/) (2019)
- [Advances in apparent conceptual physics reasoning in GPT-4](https://mlsanity.com/paper/2303.17012/) (2023)
- [Can ChatGPT pass the Vietnamese National High School Graduation Examination?](https://mlsanity.com/paper/2306.09170/) (2023)

## Built on by (most cited first, 4 of 4)

- [Large Language Model Reasoning Failures](https://mlsanity.com/paper/2602.06176/) (2026)
- [ChemSafetyBench: Benchmarking LLM Safety on Chemistry Domain](https://mlsanity.com/paper/2411.16736/) (2024)
- [PhysicsEval: Inference-Time Techniques to Improve the Reasoning Proficiency of Large Language Models on Physics Problems](https://mlsanity.com/paper/2508.00079/) (2025)
- [From Test-taking to Cognitive Scaffolding: A Pedagogical Diagnostic Benchmark for LLMs on English Standardized Tests](https://mlsanity.com/paper/2505.17056/) (2025)

## Abstract (arXiv)

With the recent developments in large language models (LLMs) and their widespread availability through open source models and/or low-cost APIs, several exciting products and applications are emerging, many of which are in the field of STEM educational technology for K-12 and university students. There is a need to evaluate these powerful language models on several benchmarks, in order to understand their risks and limitations. In this short paper, we summarize and analyze the performance of Bard, a popular LLM-based conversational service made available by Google, on the standardized Physics GRE examination.