# Your Students Don't Use LLMs Like You Wish They Did

arXiv: https://arxiv.org/abs/2604.23486 · HTML: https://mlsanity.com/paper/2604.23486/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 9 of its 56 references that are in the map.
- First posted on arXiv 26 Apr 2026.

## Builds on (most cited first, 9 of 9)

- [CodeAid: Evaluating a Classroom Deployment of an LLM-based Programming Assistant that Balances Student and Educator Needs](https://mlsanity.com/paper/2401.11314/) (2024)
- [Unifying AI Tutor Evaluation: An Evaluation Taxonomy for Pedagogical Ability Assessment of LLM-Powered AI Tutors](https://mlsanity.com/paper/2412.09416/) (2024)
- [The BEA 2023 Shared Task on Generating AI Teacher Responses in Educational Dialogues](https://mlsanity.com/paper/2306.06941/) (2023)
- [Large Language Model as an Assignment Evaluator: Insights, Feedback, and Challenges in a 1000+ Student Course](https://mlsanity.com/paper/2407.05216/) (2024)
- [Improving Socratic Question Generation using Data Augmentation and Preference Optimization](https://mlsanity.com/paper/2403.00199/) (2024)
- [The Promises and Pitfalls of Using Language Models to Measure Instruction Quality in Education](https://mlsanity.com/paper/2404.02444/) (2024)
- [RECIPE4U: Student-ChatGPT Interaction Dataset in EFL Writing Education](https://mlsanity.com/paper/2403.08272/) (2024)
- [A Scoping Study of Evaluation Practices for Responsible AI Tools: Steps Towards Effectiveness Evaluations](https://mlsanity.com/paper/2401.17486/) (2024)
- [Anna Karenina Strikes Again: Pre-Trained LLM Embeddings May Favor High-Performing Learners](https://mlsanity.com/paper/2406.06599/) (2024)

## Built on by (most cited first, 1 of 1)

- [Auditable Release Control for Pedagogical Leakage in LLM Tutors](https://mlsanity.com/paper/2608.00515/) (2026)

## Abstract (arXiv)

Educational NLP systems are typically evaluated using engagement metrics and satisfaction surveys, which are at best a proxy for meeting pedagogical goals. We introduce six computational metrics for automated evaluation of pedagogical alignment in student-AI dialogue. We validate our metrics through analysis of 12,650 messages across 500 conversations from four courses. Using our metrics, we identify a fundamental misalignment: educators design conversational tutors for sustained learning dialogue, but students mainly use them for answer-extraction. Deployment context is the strongest predictor of usage patterns, outweighing student preference or system design: when AI tools are optional, usage concentrates around deadlines; when integrated into course structure, students ask for solutions to verbatim assignment questions. Whole-dialogue evaluation misses these turn-by-turn patterns. Our metrics will enable researchers building educational dialogue systems to measure whether they are achieving their pedagogical goals.