# Do LLMs Make Mistakes Like Students? Exploring Natural Alignment between Language Models and Human Error Patterns arXiv: https://arxiv.org/abs/2502.15140 ยท HTML: https://mlsanity.com/paper/2502.15140/ ## Key facts - Cited by 8 papers in the map (data through 14 Oct 2026). - Builds on 20 of its 37 references that are in the map. - First posted on arXiv 21 Feb 2025. ## Builds on (most cited first, 20 of 20) - [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023) - [Qwen Technical Report](https://mlsanity.com/paper/2309.16609/) (2023) - [Deep Knowledge Tracing](https://mlsanity.com/paper/1506.05908/) (2015) - [MathDial: A Dialogue Tutoring Dataset with Rich Pedagogical Properties Grounded in Math Reasoning Problems](https://mlsanity.com/paper/2305.14536/) (2023) - [CLASS: A Design Framework for building Intelligent Tutoring Systems based on Learning Science principles](https://mlsanity.com/paper/2305.13272/) (2023) - [Exploring Knowledge Tracing in Tutor-Student Dialogues using LLMs](https://mlsanity.com/paper/2409.16490/) (2024) - [Stepwise Verification and Remediation of Student Reasoning Errors with Large Language Model Tutors](https://mlsanity.com/paper/2407.09136/) (2024) - [Ruffle&Riley: Insights from Designing and Evaluating a Large Language Model-Based Conversational Tutoring System](https://mlsanity.com/paper/2404.17460/) (2024) - [Automated Distractor and Feedback Generation for Math Multiple-choice Questions via In-context Learning](https://mlsanity.com/paper/2308.03234/) (2023) - [Assessing Student Errors in Experimentation Using Artificial Intelligence and Large Language Models: A Comparative Study with Human Raters](https://mlsanity.com/paper/2308.06088/) (2023) - [LLM-based Cognitive Models of Students with Misconceptions](https://mlsanity.com/paper/2410.12294/) (2024) - [Distractor generation for multiple-choice questions with predictive prompting and large language models](https://mlsanity.com/paper/2307.16338/) (2023) - [Novice Learner and Expert Tutor: Evaluating Math Reasoning Abilities of Large Language Models with Misconceptions](https://mlsanity.com/paper/2310.02439/) (2023) - [Exploring Automated Distractor Generation for Math Multiple-choice Questions via Large Language Models](https://mlsanity.com/paper/2404.02124/) (2024) - [Improving Automated Distractor Generation for Math Multiple-choice Questions with Overgenerate-and-rank](https://mlsanity.com/paper/2405.05144/) (2024) - [Generating Plausible Distractors for Multiple-Choice Questions via Student Choice Prediction](https://mlsanity.com/paper/2501.13125/) (2025) - [Data-Mining Textual Responses to Uncover Misconception Patterns](https://mlsanity.com/paper/1703.08544/) (2017) - [Panza: Design and Analysis of a Fully-Local Personalized Text Writing Assistant](https://mlsanity.com/paper/2407.10994/) (2024) - [Automated Feedback in Math Education: A Comparative Analysis of LLMs for Open-Ended Responses](https://mlsanity.com/paper/2411.08910/) (2024) - [Beyond Search Engines: Can Large Language Models Improve Curriculum Development?](https://mlsanity.com/paper/2412.07422/) (2024) ## Built on by (most cited first, 8 of 8) - [Exploring LLMs for Predicting Tutor Strategy and Student Outcomes in Dialogues](https://mlsanity.com/paper/2507.06910/) (2025) - [FoundationalASSIST: An Educational Dataset for Foundational Knowledge Tracing and Pedagogical Grounding of LLMs](https://mlsanity.com/paper/2602.00070/) (2026) - [From Test-taking to Cognitive Scaffolding: A Pedagogical Diagnostic Benchmark for LLMs on English Standardized Tests](https://mlsanity.com/paper/2505.17056/) (2025) - [BEAGLE: Behavior-Enforced Agent for Grounded Learner Emulation](https://mlsanity.com/paper/2602.13280/) (2026) - [AI-Mediated Feedback Improves Student Revisions: A Randomized Trial with FeedbackWriter in a Large Undergraduate Course](https://mlsanity.com/paper/2602.16820/) (2026) - [Rethinking Multiple-Choice Questions for RLVR: Unlocking Potential via Distractor Design](https://mlsanity.com/paper/2603.12826/) (2026) - [Assessment Design in the AI Era: A Method for Identifying Items Functioning Differentially for Humans and Chatbots](https://mlsanity.com/paper/2603.23682/) (2026) - [From Retrieval to Weights: Parametric Individualization of Small Language Models with Individual Text Corpora](https://mlsanity.com/paper/2609.10155/) (2026) ## Abstract (arXiv) Large Language Models (LLMs) have demonstrated remarkable capabilities in various educational tasks, yet their alignment with human learning patterns, particularly in predicting which incorrect options students are most likely to select in multiple-choice questions (MCQs), remains underexplored. Our work investigates the relationship between LLM generation likelihood and student response distributions in MCQs with a specific focus on distractor selections. We collect a comprehensive dataset of MCQs with real-world student response distributions to explore two fundamental research questions: (1). RQ1 - Do the distractors that students more frequently select correspond to those that LLMs assign higher generation likelihood to? (2). RQ2 - When an LLM selects a incorrect choice, does it choose the same distractor that most students pick? Our experiments reveals moderate correlations between LLM-assigned probabilities and student selection patterns for distractors in MCQs. Additionally, when LLMs make mistakes, they are more likley to select the same incorrect answers that commonly mislead students, which is a pattern consistent across both small and large language models. Our work provides empirical evidence that despite LLMs' strong performance on generating educational content, there remains a gap between LLM's underlying reasoning process and human cognitive processes in identifying confusing distractors. Our findings also have significant implications for educational assessment development. The smaller language models could be efficiently utilized for automated distractor generation as they demonstrate similar patterns in identifying confusing answer choices as larger language models. This observed alignment between LLMs and student misconception patterns opens new opportunities for generating high-quality distractors that complement traditional human-designed distractors.