mlsanityData to 14 Oct 2026

From 'F' to 'A' on the N.Y. Regents Science Exams: An Overview of the Aristo Project

arXiv 1909.01958 ↗First posted 4 Sep 2019cs.CL cs.AIcited by 33

Not on the map. Why?

Abstract (arXiv)

AI has achieved remarkable mastery over games such as Chess, Go, and Poker, and even Jeopardy, but the rich variety of standardized exams has remained a landmark challenge. Even in 2016, the best AI system achieved merely 59.3% on an 8th Grade science exam challenge. This paper reports unprecedented success on the Grade 8 New York Regents Science Exam, where for the first time a system scores more than 90% on the exam's non-diagram, multiple choice (NDMC) questions. In addition, our Aristo system, building upon the success of recent language models, exceeded 83% on the corresponding Grade 12 Science Exam NDMC questions. The results, on unseen test questions, are robust across different test years and different variations of this kind of test. They demonstrate that modern NLP methods can result in mastery on this task. While not a full solution to general question-answering (the questions are multiple choice, and the domain is restricted to 8th Grade science), it represents a significant milestone for the field.

Key facts. Cited by 33 papers in the map (data through 14 Oct 2026). First posted on arXiv 4 Sep 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 11; 2022: 9; 2023: 5; 2024: 5; 2025: 1; 2026: 2.

33 papers.

Sort:

By area:

  1. 2021QA-GNN: Reasoning with Language Models and Knowledge Graphs for Question Answeringcited by 388
  2. 2022ScienceWorld: Is your Agent Smarter than a 5th Grader?cited by 327
  3. 2023Large Language Models Sensitivity to The Order of Options in Multiple-Choice Questionscited by 263
  4. 2022GreaseLM: Graph REASoning Enhanced Language Models for Question Answeringcited by 171
  5. 2022A Survey of Deep Learning for Mathematical Reasoningcited by 124
  6. 2021Representing Numbers in NLP: a Survey and a Visioncited by 83
  7. 2021JointLK: Joint Reasoning with Language Models and Knowledge Graphs for Commonsense Question Answeringcited by 40
  8. 2022Towards Teachable Reasoning Systems: Using a Dynamic Memory of User Feedback for Continual System Improvementcited by 37
  9. 2023KnowGPT: Knowledge Graph based Prompting for Large Language Modelscited by 31
  10. 2024Evaluating LLMs' Mathematical Reasoning in Financial Document Question Answeringcited by 25
  11. 2023Mathematical Language Models: A Surveycited by 24
  12. 2021GNN is a Counter? Revisiting GNN for Question Answeringcited by 21
21 more
  1. 2021Factoring Statutory Reasoning as Language Understanding Challengescited by 20
  2. 2021Think about it! Improving defeasible reasoning by first modeling the question scenariocited by 20
  3. 2023GrapeQA: GRaph Augmentation and Pruning to Enhance Question-Answeringcited by 17
  4. 2025A Survey on Mathematical Reasoning and Optimization with Large Language Modelscited by 12
  5. 2022Leaf: Multiple-Choice Question Generationcited by 9
  6. 2022Great Truths are Always Simple: A Rather Simple Knowledge Encoder for Enhancing the Commonsense Reasoning Capacity of Pre-Trained Modelscited by 9
  7. 2024MANGO: A Benchmark for Evaluating Mapping and Navigation Abilities of Large Language Modelscited by 9
  8. 2021Deep Transfer Learning & Beyond: Transformer Language Models in Information Systems Researchcited by 7
  9. 2022Relation-Aware Language-Graph Transformer for Question Answeringcited by 7
  10. 2022Dynamic Relevance Graph Network for Knowledge-Aware Question Answeringcited by 4
  11. 2021Focusing Knowledge-based Graph Argument Mining via Topic Modelingcited by 3
  12. 2021Encoding Explanatory Knowledge for Zero-shot Science Question Answeringcited by 3
  13. 2021Inside ASCENT: Exploring a Deep Commonsense Knowledge Base and its Usage in Question Answeringcited by 3
  14. 2022Structured Knowledge Grounding for Question Answeringcited by 2
  15. 2023Teaching Smaller Language Models To Generalise To Unseen Compositional Questionscited by 2
  16. 2024PipeNet: Question Answering with Semantic Pruning over Knowledge Graphscited by 1
  17. 2024SemPool: Simple, robust, and interpretable KG pooling for enhancing language modelscited by 1
  18. 2021Ranking Facts for Explaining Answers to Elementary Science Questionscited by 0
  19. 2024Teaching Smaller Language Models To Generalise To Unseen Compositional Questions (Full Thesis)cited by 0
  20. 2026Check The Scoreboard: An Analysis of Scoring Schemes on Multiple-Choice Evaluationcited by 0
  21. 2026VisKG-LM: Compiling Knowledge Graphs into Visual Memory for Multiple-Choice Question Answeringcited by 0