# Evaluating and Optimizing Educational Content with Large Language Model Judgments

arXiv: https://arxiv.org/abs/2403.02795 · HTML: https://mlsanity.com/paper/2403.02795/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- Builds on 26 of its 56 references that are in the map.
- First posted on arXiv 5 Mar 2024.

## Builds on (most cited first, 26 of 26)

- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Generative Agents: Interactive Simulacra of Human Behavior](https://mlsanity.com/paper/2304.03442/) (2023)
- [Self-Refine: Iterative Refinement with Self-Feedback](https://mlsanity.com/paper/2303.17651/) (2023)
- [Constitutional AI: Harmlessness from AI Feedback](https://mlsanity.com/paper/2212.08073/) (2022)
- [Sparks of Artificial General Intelligence: Early experiments with GPT-4](https://mlsanity.com/paper/2303.12712/) (2023)
- [Large Language Models as Optimizers](https://mlsanity.com/paper/2309.03409/) (2023)
- [Out of One, Many: Using Language Models to Simulate Human Samples](https://mlsanity.com/paper/2209.06899/) (2022)
- [Automatic Prompt Optimization with "Gradient Descent" and Beam Search](https://mlsanity.com/paper/2305.03495/) (2023)
- [Using Large Language Models to Simulate Multiple Humans and Replicate Human Subject Studies](https://mlsanity.com/paper/2208.10264/) (2022)
- [Social Simulacra: Creating Populated Prototypes for Social Computing Systems](https://mlsanity.com/paper/2208.04024/) (2022)
- [Self-Taught Optimizer (STOP): Recursively Self-Improving Code Generation](https://mlsanity.com/paper/2310.02304/) (2023)
- [Automatic Generation of Programming Exercises and Code Explanations using Large Language Models](https://mlsanity.com/paper/2206.11861/) (2022)
- [Solving Math Word Problems by Combining Language Models With Symbolic Solvers](https://mlsanity.com/paper/2304.09102/) (2023)
- [Rehearsal: Simulating Conflict to Teach Conflict Resolution](https://mlsanity.com/paper/2309.12309/) (2023)
- [Learning gain differences between ChatGPT and human tutor generated algebra hints](https://mlsanity.com/paper/2302.06871/) (2023)
- [CGMI: Configurable General Multi-Agent Interaction Framework](https://mlsanity.com/paper/2308.12503/) (2023)
- [Teach AI How to Code: Using Large Language Models as Teachable Agents for Programming Education](https://mlsanity.com/paper/2309.14534/) (2023)
- [Is ChatGPT a Good Teacher Coach? Measuring Zero-Shot Performance For Scoring and Providing Actionable Insights on Classroom Instruction](https://mlsanity.com/paper/2306.03090/) (2023)
- [Large Language Models (GPT) for automating feedback on programming assignments](https://mlsanity.com/paper/2307.00150/) (2023)
- [Automating Human Tutor-Style Programming Feedback: Leveraging GPT-4 Tutor Model for Hint Generation and GPT-3.5 Student Model for Hint Validation](https://mlsanity.com/paper/2310.03780/) (2023)
- [Do Language Models Exhibit the Same Cognitive Biases in Problem Solving as Human Learners?](https://mlsanity.com/paper/2401.18070/) (2024)
- [Generating and Evaluating Tests for K-12 Students with Language Model Simulations: A Case Study on Sentence Reading Efficiency](https://mlsanity.com/paper/2310.06837/) (2023)
- [Improving Interpersonal Communication by Simulating Audiences with Language Models](https://mlsanity.com/paper/2311.00687/) (2023)
- [Can We Trust AI-Generated Educational Content? Comparative Analysis of Human and AI-Generated Learning Resources](https://mlsanity.com/paper/2306.10509/) (2023)
- [Ruffle&Riley: Towards the Automated Induction of Conversational Tutoring Systems](https://mlsanity.com/paper/2310.01420/) (2023)
- [Large Language Models for In-Context Student Modeling: Synthesizing Student's Behavior in Visual Programming](https://mlsanity.com/paper/2310.10690/) (2023)

## Built on by (most cited first, 12 of 12)

- [Training LLM-based Tutors to Improve Student Learning Outcomes in Dialogues](https://mlsanity.com/paper/2503.06424/) (2025)
- [EVOLvE: Evaluating and Optimizing LLMs For In-Context Exploration](https://mlsanity.com/paper/2410.06238/) (2024)
- [TeachTune: Reviewing Pedagogical Agents Against Diverse Student Profiles with Simulated Students](https://mlsanity.com/paper/2410.04078/) (2024)
- [Embracing Imperfection: Simulating Students with Diverse Cognitive Levels Using LLM-based Agents](https://mlsanity.com/paper/2505.19997/) (2025)
- [Psychometric Alignment: Capturing Human Knowledge Distributions via Language Models](https://mlsanity.com/paper/2407.15645/) (2024)
- [EduPlanner: LLM-Based Multi-Agent Systems for Customized and Intelligent Instructional Design](https://mlsanity.com/paper/2504.05370/) (2025)
- [Simulated Students in Tutoring Dialogues: Substance or Illusion?](https://mlsanity.com/paper/2601.04025/) (2026)
- [PATS: Personality-Aware Teaching Strategies with Large Language Model Tutors](https://mlsanity.com/paper/2601.08402/) (2026)
- [Which Questions Improve Learning the Most? Utility Estimation of Questions with LM-based Simulations](https://mlsanity.com/paper/2502.17383/) (2025)
- [The Impact of Item-Writing Flaws on Difficulty and Discrimination in Item Response Theory](https://mlsanity.com/paper/2503.10533/) (2025)
- [From Tool to Teammate: LLM Coding Agents as Collaborative Partners for Behavioral Labeling in Educational Dialogue Analysis](https://mlsanity.com/paper/2603.27440/) (2026)
- [Effective Personalized AI Tutors via LLM-Guided Reinforcement Learning](https://mlsanity.com/paper/2608.16907/) (2026)

## Abstract (arXiv)

Creating effective educational materials generally requires expensive and time-consuming studies of student learning outcomes. To overcome this barrier, one idea is to build computational models of student learning and use them to optimize instructional materials. However, it is difficult to model the cognitive processes of learning dynamics. We propose an alternative approach that uses Language Models (LMs) as educational experts to assess the impact of various instructions on learning outcomes. Specifically, we use GPT-3.5 to evaluate the overall effect of instructional materials on different student groups and find that it can replicate well-established educational findings such as the Expertise Reversal Effect and the Variability Effect. This demonstrates the potential of LMs as reliable evaluators of educational content. Building on this insight, we introduce an instruction optimization approach in which one LM generates instructional materials using the judgments of another LM as a reward function. We apply this approach to create math word problem worksheets aimed at maximizing student learning gains. Human teachers' evaluations of these LM-generated worksheets show a significant alignment between the LM judgments and human teacher preferences. We conclude by discussing potential divergences between human and LM opinions and the resulting pitfalls of automating instructional design.