# Grading Massive Open Online Courses Using Large Language Models

arXiv: https://arxiv.org/abs/2406.11102 · HTML: https://mlsanity.com/paper/2406.11102/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 35 references that are in the map.
- First posted on arXiv 16 Jun 2024.

## Builds on (most cited first, 17 of 17)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Large Language Models are Zero-Shot Reasoners](https://mlsanity.com/paper/2205.11916/) (2022)
- [G-Eval: NLG Evaluation using GPT-4 with Better Human Alignment](https://mlsanity.com/paper/2303.16634/) (2023)
- [Large Language Models Are State-of-the-Art Evaluators of Translation Quality](https://mlsanity.com/paper/2302.14520/) (2023)
- [JudgeLM: Fine-tuned Large Language Models are Scalable Judges](https://mlsanity.com/paper/2310.17631/) (2023)
- [Evaluating Large Language Models at Evaluating Instruction Following](https://mlsanity.com/paper/2310.07641/) (2023)
- [ChatGPT as a Factual Inconsistency Evaluator for Text Summarization](https://mlsanity.com/paper/2303.15621/) (2023)
- [Exploring LLM Prompting Strategies for Joint Essay Scoring and Feedback Generation](https://mlsanity.com/paper/2404.15845/) (2024)
- [Large Language Models (GPT) for automating feedback on programming assignments](https://mlsanity.com/paper/2307.00150/) (2023)
- [Exploring the Potential of Large Language Models to Generate Formative Programming Feedback](https://mlsanity.com/paper/2309.00029/) (2023)
- [A large language model-assisted education tool to provide feedback on open-ended responses](https://mlsanity.com/paper/2308.02439/) (2023)
- [Large Language Models for Education: Grading Open-Ended Questions Using ChatGPT](https://mlsanity.com/paper/2307.16696/) (2023)
- [Evaluating the Application of Large Language Models to Generate Feedback in Programming Education](https://mlsanity.com/paper/2403.09744/) (2024)
- [Balancing Cost and Quality: An Exploration of Human-in-the-loop Frameworks for Automated Short Answer Scoring](https://mlsanity.com/paper/2206.08288/) (2022)
- [Students' Perceptions and Preferences of Generative Artificial Intelligence Feedback for Programming](https://mlsanity.com/paper/2312.11567/) (2023)

## Built on by (most cited first, 7 of 7)

- [CoTAL: Human-in-the-Loop Prompt Engineering for Generalizable Formative Assessment Scoring and Feedback](https://mlsanity.com/paper/2504.02323/) (2025)
- [Intent Laundering: AI Safety Datasets Are Not What They Seem](https://mlsanity.com/paper/2602.16729/) (2026)
- [Teaching Introduction to Programming in the times of AI: A case study of a course re-design](https://mlsanity.com/paper/2508.06572/) (2025)
- [Using Contextually Aligned Online Reviews to Measure LLMs' Performance Disparities Across Language Varieties](https://mlsanity.com/paper/2502.07058/) (2025)
- [LLM-as-a-Grader: Practical Insights from Large Language Model for Short-Answer and Report Evaluation](https://mlsanity.com/paper/2511.10819/) (2025)
- [Empirical Prompt Engineering for Construct Identification with Large Language Models](https://mlsanity.com/paper/2512.03818/) (2025)
- [AI-Enabled grading with near-domain data for scaling feedback with human-level accuracy](https://mlsanity.com/paper/2512.04113/) (2025)

## Abstract (arXiv)

Massive open online courses (MOOCs) offer free education globally. Despite this democratization of learning, the massive enrollment in these courses makes it impractical for an instructor to assess every student's writing assignment. As a result, peer grading, often guided by a straightforward rubric, is the method of choice. While convenient, peer grading often falls short in terms of reliability and validity. In this study, we explore the feasibility of using large language models (LLMs) to replace peer grading in MOOCs. To this end, we adapt the zero-shot chain-of-thought (ZCoT) prompting technique to automate the feedback process once the LLM assigns a score to an assignment. Specifically, to instruct LLMs for grading, we use three distinct prompts based on ZCoT: (1) ZCoT with instructor-provided correct answers, (2) ZCoT with both instructor-provided correct answers and rubrics, and (3) ZCoT with instructor-provided correct answers and LLM-generated rubrics. We tested these prompts in 18 different scenarios using two LLMs, GPT-4 and GPT-3.5, across three MOOCs: Introductory Astronomy, Astrobiology, and the History and Philosophy of Astronomy. Our results show that ZCoT, when augmented with instructor-provided correct answers and rubrics, produces grades that are more aligned with those assigned by instructors compared to peer grading. Finally, our findings indicate a promising potential for automated grading systems in MOOCs, especially in subjects with well-defined rubrics, to improve the learning experience for millions of online learners worldwide.