# Evaluating Large Language Models in Analysing Classroom Dialogue

arXiv: https://arxiv.org/abs/2402.02380 · HTML: https://mlsanity.com/paper/2402.02380/

## Key facts

- Cited by 19 papers in the map (data through 14 Oct 2026).
- Builds on 7 of its 68 references that are in the map.
- First posted on arXiv 4 Feb 2024.

## Builds on (most cited first, 7 of 7)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [On the Opportunities and Risks of Foundation Models](https://mlsanity.com/paper/2108.07258/) (2021)
- [The TalkMoves Dataset: K-12 Mathematics Lesson Transcripts Annotated for Teacher and Student Discursive Moves](https://mlsanity.com/paper/2204.09652/) (2022)
- [Computationally Identifying Funneling and Focusing Questions in Classroom Discourse](https://mlsanity.com/paper/2208.04715/) (2022)

## Built on by (most cited first, 19 of 19)

- [StelLA: Subspace Learning in Low-rank Adaptation using Stiefel Manifold](https://mlsanity.com/paper/2510.01938/) (2025)
- [A validity-guided workflow for robust large language model research in psychology](https://mlsanity.com/paper/2507.04491/) (2025)
- [EduEval: A Hierarchical Cognitive Benchmark for Evaluating Large Language Models in Chinese Education](https://mlsanity.com/paper/2512.00290/) (2025)
- [How well do Large Language Models Recognize Instructional Moves? Establishing Baselines for Foundation Models in Educational Discourse](https://mlsanity.com/paper/2512.19903/) (2025)
- [RAM2C: A Liberal Arts Educational Chatbot based on Retrieval-augmented Multi-role Multi-expert Collaboration](https://mlsanity.com/paper/2409.15461/) (2024)
- [Combining physics education and machine learning research to measure evidence of students' mechanistic sensemaking](https://mlsanity.com/paper/2503.15638/) (2025)
- [OnRL-RAG: Real-Time Personalized Mental Health Dialogue System](https://mlsanity.com/paper/2504.02894/) (2025)
- [Exploring Automated Recognition of Instructional Activity and Discourse from Multimodal Classroom Data](https://mlsanity.com/paper/2512.00087/) (2025)
- [Sandpiper: Orchestrated AI-Annotation for Educational Discourse at Scale](https://mlsanity.com/paper/2603.08406/) (2026)
- [EduCoder: An Open-Source Annotation System for Education Transcript Data](https://mlsanity.com/paper/2507.05385/) (2025)
- [Signals Are Not States: Neuro-Symbolic Safeguards for Culturally Aware Classroom AI](https://mlsanity.com/paper/2603.22793/) (2026)
- [When AI Meets Early Childhood Education: Large Language Models as Assessment Teammates in Chinese Preschools](https://mlsanity.com/paper/2603.24389/) (2026)
- [Mitigating LLM biases toward spurious social contexts using direct preference optimization](https://mlsanity.com/paper/2604.02585/) (2026)
- [Can LLMs Reason About Attention? Towards Zero-Shot Analysis of Multimodal Classroom Behavior](https://mlsanity.com/paper/2604.03401/) (2026)
- [Locating acts of mechanistic reasoning in student team conversations with mechanistic machine learning](https://mlsanity.com/paper/2604.21870/) (2026)
- [Controllable Spoken Dialogue Generation: An LLM-Driven Grading System for K-12 Non-Native English Learners](https://mlsanity.com/paper/2604.22542/) (2026)
- [Mapping the Methodological Space of Classroom Interaction Research: Scale, Duration, and Modality in an Age of AI](https://mlsanity.com/paper/2604.28098/) (2026)
- [Walking through Discussions: A Mobile Visual Analytics System for In-Situ Group Discussion Analysis](https://mlsanity.com/paper/2608.08617/) (2026)
- [EduBehaviors: Assertion-based Schemas for Auditable Coding of Educational Dialogues](https://mlsanity.com/paper/2609.27043/) (2026)

## Abstract (arXiv)

This study explores the application of Large Language Models (LLMs), specifically GPT-4, in the analysis of classroom dialogue, a crucial research task for both teaching diagnosis and quality improvement. Recognizing the knowledge-intensive and labor-intensive nature of traditional qualitative methods in educational research, this study investigates the potential of LLM to streamline and enhance the analysis process. The study involves datasets from a middle school, encompassing classroom dialogues across mathematics and Chinese classes. These dialogues were manually coded by educational experts and then analyzed using a customised GPT-4 model. This study focuses on comparing manual annotations with the outputs of GPT-4 to evaluate its efficacy in analyzing educational dialogues. Time efficiency, inter-coder agreement, and inter-coder reliability between human coders and GPT-4 are evaluated. Results indicate substantial time savings with GPT-4, and a high degree of consistency in coding between the model and human coders, with some discrepancies in specific codes. These findings highlight the strong potential of LLM in teaching evaluation and facilitation.