# From 'F' to 'A' on the N.Y. Regents Science Exams: An Overview of the Aristo Project

arXiv: https://arxiv.org/abs/1909.01958 · HTML: https://mlsanity.com/paper/1909.01958/

## Key facts

- Cited by 33 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 4 Sep 2019.

## Built on by (most cited first, 30 of 33)

- [QA-GNN: Reasoning with Language Models and Knowledge Graphs for Question Answering](https://mlsanity.com/paper/2104.06378/) (2021)
- [ScienceWorld: Is your Agent Smarter than a 5th Grader?](https://mlsanity.com/paper/2203.07540/) (2022)
- [Large Language Models Sensitivity to The Order of Options in Multiple-Choice Questions](https://mlsanity.com/paper/2308.11483/) (2023)
- [GreaseLM: Graph REASoning Enhanced Language Models for Question Answering](https://mlsanity.com/paper/2201.08860/) (2022)
- [A Survey of Deep Learning for Mathematical Reasoning](https://mlsanity.com/paper/2212.10535/) (2022)
- [Representing Numbers in NLP: a Survey and a Vision](https://mlsanity.com/paper/2103.13136/) (2021)
- [JointLK: Joint Reasoning with Language Models and Knowledge Graphs for Commonsense Question Answering](https://mlsanity.com/paper/2112.02732/) (2021)
- [Towards Teachable Reasoning Systems: Using a Dynamic Memory of User Feedback for Continual System Improvement](https://mlsanity.com/paper/2204.13074/) (2022)
- [KnowGPT: Knowledge Graph based Prompting for Large Language Models](https://mlsanity.com/paper/2312.06185/) (2023)
- [Evaluating LLMs' Mathematical Reasoning in Financial Document Question Answering](https://mlsanity.com/paper/2402.11194/) (2024)
- [Mathematical Language Models: A Survey](https://mlsanity.com/paper/2312.07622/) (2023)
- [GNN is a Counter? Revisiting GNN for Question Answering](https://mlsanity.com/paper/2110.03192/) (2021)
- [Factoring Statutory Reasoning as Language Understanding Challenges](https://mlsanity.com/paper/2105.07903/) (2021)
- [Think about it! Improving defeasible reasoning by first modeling the question scenario](https://mlsanity.com/paper/2110.12349/) (2021)
- [GrapeQA: GRaph Augmentation and Pruning to Enhance Question-Answering](https://mlsanity.com/paper/2303.12320/) (2023)
- [A Survey on Mathematical Reasoning and Optimization with Large Language Models](https://mlsanity.com/paper/2503.17726/) (2025)
- [Leaf: Multiple-Choice Question Generation](https://mlsanity.com/paper/2201.09012/) (2022)
- [Great Truths are Always Simple: A Rather Simple Knowledge Encoder for Enhancing the Commonsense Reasoning Capacity of Pre-Trained Models](https://mlsanity.com/paper/2205.01841/) (2022)
- [MANGO: A Benchmark for Evaluating Mapping and Navigation Abilities of Large Language Models](https://mlsanity.com/paper/2403.19913/) (2024)
- [Deep Transfer Learning & Beyond: Transformer Language Models in Information Systems Research](https://mlsanity.com/paper/2110.08975/) (2021)
- [Relation-Aware Language-Graph Transformer for Question Answering](https://mlsanity.com/paper/2212.00975/) (2022)
- [Dynamic Relevance Graph Network for Knowledge-Aware Question Answering](https://mlsanity.com/paper/2209.09947/) (2022)
- [Focusing Knowledge-based Graph Argument Mining via Topic Modeling](https://mlsanity.com/paper/2102.02086/) (2021)
- [Encoding Explanatory Knowledge for Zero-shot Science Question Answering](https://mlsanity.com/paper/2105.05737/) (2021)
- [Inside ASCENT: Exploring a Deep Commonsense Knowledge Base and its Usage in Question Answering](https://mlsanity.com/paper/2105.13662/) (2021)
- [Structured Knowledge Grounding for Question Answering](https://mlsanity.com/paper/2209.08284/) (2022)
- [Teaching Smaller Language Models To Generalise To Unseen Compositional Questions](https://mlsanity.com/paper/2308.00946/) (2023)
- [PipeNet: Question Answering with Semantic Pruning over Knowledge Graphs](https://mlsanity.com/paper/2401.17536/) (2024)
- [SemPool: Simple, robust, and interpretable KG pooling for enhancing language models](https://mlsanity.com/paper/2402.02289/) (2024)
- [Ranking Facts for Explaining Answers to Elementary Science Questions](https://mlsanity.com/paper/2110.09036/) (2021)

## Abstract (arXiv)

AI has achieved remarkable mastery over games such as Chess, Go, and Poker, and even Jeopardy, but the rich variety of standardized exams has remained a landmark challenge. Even in 2016, the best AI system achieved merely 59.3% on an 8th Grade science exam challenge. This paper reports unprecedented success on the Grade 8 New York Regents Science Exam, where for the first time a system scores more than 90% on the exam's non-diagram, multiple choice (NDMC) questions. In addition, our Aristo system, building upon the success of recent language models, exceeded 83% on the corresponding Grade 12 Science Exam NDMC questions. The results, on unseen test questions, are robust across different test years and different variations of this kind of test. They demonstrate that modern NLP methods can result in mastery on this task. While not a full solution to general question-answering (the questions are multiple choice, and the domain is restricted to 8th Grade science), it represents a significant milestone for the field.