# Findings of the BEA 2025 Shared Task on Pedagogical Ability Assessment of AI-powered Tutors arXiv: https://arxiv.org/abs/2507.10579 ยท HTML: https://mlsanity.com/paper/2507.10579/ ## Key facts - Cited by 18 papers in the map (data through 14 Oct 2026). - Builds on 15 of its 56 references that are in the map. - First posted on arXiv 11 Jul 2025. ## Builds on (most cited first, 15 of 15) - [G-Eval: NLG Evaluation using GPT-4 with Better Human Alignment](https://mlsanity.com/paper/2303.16634/) (2023) - [A Call for Clarity in Reporting BLEU Scores](https://mlsanity.com/paper/1804.08771/) (2018) - [ChatGLM: A Family of Large Language Models from GLM-130B to GLM-4 All Tools](https://mlsanity.com/paper/2406.12793/) (2024) - [MPNet: Masked and Permuted Pre-training for Language Understanding](https://mlsanity.com/paper/2004.09297/) (2020) - [MathDial: A Dialogue Tutoring Dataset with Rich Pedagogical Properties Grounded in Math Reasoning Problems](https://mlsanity.com/paper/2305.14536/) (2023) - [The AI Teacher Test: Measuring the Pedagogical Ability of Blender and GPT-3 in Educational Dialogues](https://mlsanity.com/paper/2205.07540/) (2022) - [Dialogue Response Ranking Training with Large-Scale Human Feedback Data](https://mlsanity.com/paper/2009.06978/) (2020) - [Unifying AI Tutor Evaluation: An Evaluation Taxonomy for Pedagogical Ability Assessment of LLM-Powered AI Tutors](https://mlsanity.com/paper/2412.09416/) (2024) - [Bridging the Novice-Expert Gap via Models of Decision-Making: A Case Study on Remediating Math Mistakes](https://mlsanity.com/paper/2310.10648/) (2023) - [Stepwise Verification and Remediation of Student Reasoning Errors with Large Language Model Tutors](https://mlsanity.com/paper/2407.09136/) (2024) - [The BEA 2023 Shared Task on Generating AI Teacher Responses in Educational Dialogues](https://mlsanity.com/paper/2306.06941/) (2023) - [MSA at BEA 2025 Shared Task: Disagreement-Aware Instruction Tuning for Multi-Dimensional Evaluation of LLMs as Math Tutors](https://mlsanity.com/paper/2505.18549/) (2025) - [RETUYT-INCO at BEA 2025 Shared Task: How Far Can Lightweight Models Go in AI-powered Tutor Evaluation?](https://mlsanity.com/paper/2506.11243/) (2025) - [BD at BEA 2025 Shared Task: MPNet Ensembles for Pedagogical Mistake Identification and Localization in AI Tutor Responses](https://mlsanity.com/paper/2506.01817/) (2025) - [NeuralNexus at BEA 2025 Shared Task: Retrieval-Augmented Prompting for Mistake Identification in AI Tutors](https://mlsanity.com/paper/2506.10627/) (2025) ## Built on by (most cited first, 18 of 18) - [Opportunities and Challenges of LLMs in Education: An NLP Perspective](https://mlsanity.com/paper/2507.22753/) (2025) - [EduEVAL-DB: A Role-Based Dataset for Pedagogical Risk Evaluation in Educational Explanations](https://mlsanity.com/paper/2602.15531/) (2026) - [AIriskEval-edu: New Dataset for Risk Assessment in AI-mediated K-12 Educational Explanations](https://mlsanity.com/paper/2607.01934/) (2026) - [Beyond Helpfulness: A Teaching-over-Solving Diagnostic for Measuring Educational Impact in LLM Tutors](https://mlsanity.com/paper/2606.16206/) (2026) - [AITutor-EvalKit: Exploring the Capabilities of AI Tutors](https://mlsanity.com/paper/2512.03688/) (2025) - [ConvoLearn: A Learning Sciences Grounded Dataset for Fine-Tuning Dialogic AI Tutors](https://mlsanity.com/paper/2601.08950/) (2026) - [The Aftermath of DrawEduMath: Vision Language Models Underperform with Struggling Students and Misdiagnose Errors](https://mlsanity.com/paper/2603.00925/) (2026) - [AFRILANGTUTOR: Advancing Language Tutoring and Culture Education in Low-Resource Languages with Large Language Models](https://mlsanity.com/paper/2604.20996/) (2026) - [Large Language Models Approach Expert Pedagogical Quality in Math Tutoring but Differ in Instructional and Linguistic Profiles](https://mlsanity.com/paper/2512.20780/) (2025) - [TeachBench: A Syllabus-Grounded Framework for Evaluating Teaching Ability in Large Language Models](https://mlsanity.com/paper/2601.21375/) (2026) - [Towards Reward Modeling for AI Tutors in Math Mistake Remediation](https://mlsanity.com/paper/2603.24375/) (2026) - [BLADE: Better Language Answers through Dialogue and Explanations](https://mlsanity.com/paper/2604.03236/) (2026) - [CAMO: A Class-Aware Minority-Optimized Ensemble for Robust Language Model Evaluation on Imbalanced Data](https://mlsanity.com/paper/2604.07583/) (2026) - [Incentives Of EdTech: A Systematic Review Of EduNLP Research](https://mlsanity.com/paper/2606.13691/) (2026) - [Towards Pedagogically Aligned LLM Tutors for Math Mistake Remediation](https://mlsanity.com/paper/2606.21502/) (2026) - [Knowledge Distillation for Automated AI Tutor Evaluation](https://mlsanity.com/paper/2607.10647/) (2026) - [Do LLM Tutors Actually Teach? A Process-Centered Synthesis of the Tutoring Loop](https://mlsanity.com/paper/2610.15692/) (2026) - [A Unified Response Framework for Questions and Tutoring Dialogues](https://mlsanity.com/paper/2610.19532/) (2026) ## Abstract (arXiv) This shared task has aimed to assess pedagogical abilities of AI tutors powered by large language models (LLMs), focusing on evaluating the quality of tutor responses aimed at student's mistake remediation within educational dialogues. The task consisted of five tracks designed to automatically evaluate the AI tutor's performance across key dimensions of mistake identification, precise location of the mistake, providing guidance, and feedback actionability, grounded in learning science principles that define good and effective tutor responses, as well as the track focusing on detection of the tutor identity. The task attracted over 50 international teams across all tracks. The submitted models were evaluated against gold-standard human annotations, and the results, while promising, show that there is still significant room for improvement in this domain: the best results for the four pedagogical ability assessment tracks range between macro F1 scores of 58.34 (for providing guidance) and 71.81 (for mistake identification) on three-class problems, with the best F1 score in the tutor identification track reaching 96.98 on a 9-class task. In this paper, we overview the main findings of the shared task, discuss the approaches taken by the teams, and analyze their performance. All resources associated with this task are made publicly available to support future research in this critical domain.