# Towards a Benchmark for Scientific Understanding in Humans and Machines

arXiv: https://arxiv.org/abs/2304.10327 · HTML: https://mlsanity.com/paper/2304.10327/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Builds on 15 of its 68 references that are in the map.
- First posted on arXiv 20 Apr 2023.

## Builds on (most cited first, 15 of 15)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding](https://mlsanity.com/paper/1804.07461/) (2018)
- [Sparks of Artificial General Intelligence: Early experiments with GPT-4](https://mlsanity.com/paper/2303.12712/) (2023)
- [Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models](https://mlsanity.com/paper/2206.04615/) (2022)
- [Discovering Language Model Behaviors with Model-Written Evaluations](https://mlsanity.com/paper/2212.09251/) (2022)
- [Galactica: A Large Language Model for Science](https://mlsanity.com/paper/2211.09085/) (2022)
- [Prompt Programming for Large Language Models: Beyond the Few-Shot Paradigm](https://mlsanity.com/paper/2102.07350/) (2021)
- [Deep Learning: A Critical Appraisal](https://mlsanity.com/paper/1801.00631/) (2018)
- [Learning to Ask: Neural Question Generation for Reading Comprehension](https://mlsanity.com/paper/1705.00106/) (2017)
- [Learning to Ask Good Questions: Ranking Clarification Questions using Neural Expected Value of Perfect Information](https://mlsanity.com/paper/1805.04655/) (2018)
- [On scientific understanding with artificial intelligence](https://mlsanity.com/paper/2204.01467/) (2022)
- [Scientific Machine Learning Benchmarks](https://mlsanity.com/paper/2110.12773/) (2021)
- [AI and the FCI: Can ChatGPT Project an Understanding of Introductory Physics?](https://mlsanity.com/paper/2303.01067/) (2023)
- [SAIBench: Benchmarking AI for Science](https://mlsanity.com/paper/2206.05418/) (2022)

## Built on by (most cited first, 9 of 9)

- [Generating Interesting Scientific Ideas using Knowledge Graphs and LLMs: Evaluations with 100 Research Group Leaders](https://mlsanity.com/paper/2405.17044/) (2024)
- [Large Physics Models: Towards a collaborative approach with Large Language Models and Foundation Models](https://mlsanity.com/paper/2501.05382/) (2025)
- [Interpretable Machine Learning in Physics: A Review](https://mlsanity.com/paper/2503.23616/) (2025)
- [Meta-Designing Quantum Experiments with Language Models](https://mlsanity.com/paper/2406.02470/) (2024)
- [Reinforcement Learning from Human Feedback: Whose Culture, Whose Values, Whose Perspectives?](https://mlsanity.com/paper/2407.17482/) (2024)
- [Towards autonomous quantum physics research using LLM agents with access to intelligent tools](https://mlsanity.com/paper/2511.11752/) (2025)
- [Towards a Large Physics Benchmark](https://mlsanity.com/paper/2507.21695/) (2025)
- [Causes in neuron diagrams, and testing causal reasoning in Large Language Models. A glimpse of the future of philosophy?](https://mlsanity.com/paper/2506.14239/) (2025)
- [When benchmark inferences do not compose: Projectibility in AI evaluation](https://mlsanity.com/paper/2607.26159/) (2026)

## Abstract (arXiv)

Scientific understanding is a fundamental goal of science, allowing us to explain the world. There is currently no good way to measure the scientific understanding of agents, whether these be humans or Artificial Intelligence systems. Without a clear benchmark, it is challenging to evaluate and compare different levels of and approaches to scientific understanding. In this Roadmap, we propose a framework to create a benchmark for scientific understanding, utilizing tools from philosophy of science. We adopt a behavioral notion according to which genuine understanding should be recognized as an ability to perform certain tasks. We extend this notion by considering a set of questions that can gauge different levels of scientific understanding, covering information retrieval, the capability to arrange information to produce an explanation, and the ability to infer how things would be different under different circumstances. The Scientific Understanding Benchmark (SUB), which is formed by a set of these tests, allows for the evaluation and comparison of different approaches. Benchmarking plays a crucial role in establishing trust, ensuring quality control, and providing a basis for performance evaluation. By aligning machine and human scientific understanding we can improve their utility, ultimately advancing scientific understanding and helping to discover new insights within machines.