# Beyond Benchmarking: Scenario-Based Evaluation of Large Language Models for Personalized Learning

arXiv: https://arxiv.org/abs/2509.05346 · HTML: https://mlsanity.com/paper/2509.05346/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 6 of its 32 references that are in the map.
- First posted on arXiv 2 Sep 2025.

## Builds on (most cited first, 6 of 6)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://mlsanity.com/paper/2306.05685/) (2023)
- [UMAP: Uniform Manifold Approximation and Projection for Dimension Reduction](https://mlsanity.com/paper/1802.03426/) (2018)
- [Learning-by-teaching with ChatGPT: The effect of teachable ChatGPT agent on programming education](https://mlsanity.com/paper/2412.15226/) (2024)
- [Generative AI as a Tool for Enhancing Reflective Learning in Students](https://mlsanity.com/paper/2412.02603/) (2024)

## Built on by (most cited first, 1 of 1)

- [Rich Insights from Cheap Signals: Efficient Evaluations via Tensor Factorization](https://mlsanity.com/paper/2603.02029/) (2026)

## Abstract (arXiv)

While large language models (LLMs) are increasingly being adopted to support personalized learning, there remains limited understanding of how their pedagogical behaviors differ in authentic learning scenarios. Existing evaluation practices often emphasize benchmark scores and overall model rankings, but such approaches usually provide limited insight into how LLMs diagnose student understanding and generate personalized guidance. This study proposes a scenario-based evaluation framework for closely examining LLM behavior in personalized learning support. Using a post-class tutoring setting as an illustrative example, a dataset comprising a student's responses to a set of data structures questions is provided to multiple LLMs. Each model is required to identify the underlying knowledge concepts, infer the student's mastery profile, and generate personalized guidance for improvement. To support consistent, reproducible and scalable comparison, Gemini is employed as an external evaluator across multiple pedagogically relevant dimensions, including diagnostic accuracy, instructional clarity, actionability, misconception identification, and appropriateness to the student's level. The resulting pairwise preferences are then fitted using the Bradley-Terry model to derive comparative strength estimates, while qualitative analysis and semantic visualization are used to further examine differences in feedback structure, diagnostic depth, and recommendation specificity. The key findings show that different LLMs exhibit distinguishable pedagogical behaviors within the same learning scenario and demonstrates how scenario-based evaluation can provide educationally meaningful signals for understanding model behavior in AI-enhanced education.