# From Human Days to Machine Seconds: Automatically Answering and Generating Machine Learning Final Exams

arXiv: https://arxiv.org/abs/2206.05442 · HTML: https://mlsanity.com/paper/2206.05442/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 14 of its 24 references that are in the map.
- First posted on arXiv 11 Jun 2022.

## Builds on (most cited first, 14 of 14)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [Evaluating Large Language Models Trained on Code](https://mlsanity.com/paper/2107.03374/) (2021)
- [Measuring Massive Multitask Language Understanding](https://mlsanity.com/paper/2009.03300/) (2020)
- [Self-Consistency Improves Chain of Thought Reasoning in Language Models](https://mlsanity.com/paper/2203.11171/) (2022)
- [PaLM: Scaling Language Modeling with Pathways](https://mlsanity.com/paper/2204.02311/) (2022)
- [Large Language Models are Zero-Shot Reasoners](https://mlsanity.com/paper/2205.11916/) (2022)
- [Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models](https://mlsanity.com/paper/2206.04615/) (2022)
- [Competition-Level Code Generation with AlphaCode](https://mlsanity.com/paper/2203.07814/) (2022)
- [Least-to-Most Prompting Enables Complex Reasoning in Large Language Models](https://mlsanity.com/paper/2205.10625/) (2022)
- [Scaling Language Models: Methods, Analysis & Insights from Training Gopher](https://mlsanity.com/paper/2112.11446/) (2021)
- [A Neural Network Solves, Explains, and Generates University Math Problems by Program Synthesis and Few-Shot Learning at Human Level](https://mlsanity.com/paper/2112.15594/) (2021)
- [Solving Machine Learning Problems](https://mlsanity.com/paper/2107.01238/) (2021)
- [Asking Questions Like Educational Experts: Automatically Generating Question-Answer Pairs on Real-World Examination Data](https://mlsanity.com/paper/2109.05179/) (2021)

## Built on by (most cited first, 6 of 6)

- [ARB: Advanced Reasoning Benchmark for Large Language Models](https://mlsanity.com/paper/2307.13692/) (2023)
- [T-RAG: Lessons from the LLM Trenches](https://mlsanity.com/paper/2402.07483/) (2024)
- [Large Language Models in Computer Science Education: A Systematic Literature Review](https://mlsanity.com/paper/2410.16349/) (2024)
- [Exploring the MIT Mathematics and EECS Curriculum Using Large Language Models](https://mlsanity.com/paper/2306.08997/) (2023)
- [HardML: A Benchmark For Evaluating Data Science And Machine Learning knowledge and reasoning in AI](https://mlsanity.com/paper/2501.15627/) (2025)
- [Chat-of-Thought: Collaborative Multi-Agent System for Generating Domain Specific Information](https://mlsanity.com/paper/2506.10086/) (2025)

## Abstract (arXiv)

A final exam in machine learning at a top institution such as MIT, Harvard, or Cornell typically takes faculty days to write, and students hours to solve. We demonstrate that large language models pass machine learning finals at a human level, on finals available online after the models were trained, and automatically generate new human-quality final exam questions in seconds. Previous work has developed program synthesis and few-shot learning methods to solve university-level problem set questions in mathematics and STEM courses. In this work, we develop and compare methods that solve final exams, which differ from problem sets in several ways: the questions are longer, have multiple parts, are more complicated, and span a broader set of topics. We curate a dataset and benchmark of questions from machine learning final exams available online and code for answering these questions and generating new questions. We show how to generate new questions from other questions and course notes. For reproducibility and future research on this final exam benchmark, we use automatic checkers for multiple-choice, numeric, and questions with expression answers. We perform ablation studies comparing zero-shot learning with few-shot learning and chain-of-thought prompting using GPT-3, OPT, Codex, and ChatGPT across machine learning topics and find that few-shot learning methods perform best. We highlight the transformative potential of language models to streamline the writing and solution of large-scale assessments, significantly reducing the workload from human days to mere machine seconds. Our results suggest that rather than banning large language models such as ChatGPT in class, instructors should teach students to harness them by asking students meta-questions about correctness, completeness, and originality of the responses generated, encouraging critical thinking in academic studies.