# Auto-Evaluation: A Critical Measure in Driving Improvements in Quality and Safety of AI-Generated Lesson Resources

arXiv: https://arxiv.org/abs/2502.10410 · HTML: https://mlsanity.com/paper/2502.10410/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 23 Jan 2025.

## Built on by (most cited first, 4 of 4)

- [Benchmarking the Pedagogical Knowledge of Large Language Models](https://mlsanity.com/paper/2506.18710/) (2025)
- [Building Effective Safety Guardrails in AI Education Tools](https://mlsanity.com/paper/2508.05360/) (2025)
- [Judging the Judges: Human Validation of Multi-LLM Evaluation for High-Quality K--12 Science Instructional Materials](https://mlsanity.com/paper/2602.13243/) (2026)
- [SciEval: A Benchmark for Automatic Evaluation of K-12 Science Instructional Materials](https://mlsanity.com/paper/2604.25472/) (2026)

## Abstract (arXiv)

As a publicly funded body in the UK, Oak National Academy is in a unique position to innovate within this field as we have a comprehensive curriculum of approximately 13,000 open education resources (OER) for all National Curriculum subjects, designed and quality-assured by expert, human teachers. This has provided the corpus of content needed for building a high-quality AI-powered lesson planning tool, Aila, that is free to use and, therefore, accessible to all teachers across the country. Furthermore, using our evidence-informed curriculum principles, we have codified and exemplified each component of lesson design. To assess the quality of lessons produced by Aila at scale, we have developed an AI-powered auto-evaluation agent,facilitating informed improvements to enhance output quality. Through comparisons between human and auto-evaluations, we have begun to refine this agent further to increase its accuracy, measured by its alignment with an expert human evaluator. In this paper we present this iterative evaluation process through an illustrative case study focused on one quality benchmark - the level of challenge within multiple-choice quizzes. We also explore the contribution that this may make to similar projects and the wider sector.