# Multimodal Assessment of Classroom Discourse Quality: A Text-Centered Attention-Based Multi-Task Learning Approach

arXiv: https://arxiv.org/abs/2505.07902 · HTML: https://mlsanity.com/paper/2505.07902/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 21 of its 52 references that are in the map.
- First posted on arXiv 12 May 2025.

## Builds on (most cited first, 21 of 21)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations](https://mlsanity.com/paper/2006.11477/) (2020)
- [MTEB: Massive Text Embedding Benchmark](https://mlsanity.com/paper/2210.07316/) (2022)
- [Unsupervised Cross-lingual Representation Learning for Speech Recognition](https://mlsanity.com/paper/2006.13979/) (2020)
- [Emotion Recognition from Speech Using Wav2vec 2.0 Embeddings](https://mlsanity.com/paper/2104.03502/) (2021)
- [Measuring Conversational Uptake: A Case Study on Student-Teacher Interactions](https://mlsanity.com/paper/2106.03873/) (2021)
- [The TalkMoves Dataset: K-12 Mathematics Lesson Transcripts Annotated for Teacher and Student Discursive Moves](https://mlsanity.com/paper/2204.09652/) (2022)
- [Is ChatGPT a Good Teacher Coach? Measuring Zero-Shot Performance For Scoring and Providing Actionable Insights on Classroom Instruction](https://mlsanity.com/paper/2306.03090/) (2023)
- [Multimodal Engagement Analysis from Facial Videos in the Classroom](https://mlsanity.com/paper/2101.04215/) (2021)
- [Multimodal Multi-loss Fusion Network for Sentiment Analysis](https://mlsanity.com/paper/2308.00264/) (2023)
- [Multi-Task Contrastive Learning for 8192-Token Bilingual Text Embeddings](https://mlsanity.com/paper/2402.17016/) (2024)
- [Computationally Identifying Funneling and Focusing Questions in Classroom Discourse](https://mlsanity.com/paper/2208.04715/) (2022)
- [Toward Automated Classroom Observation: Multimodal Machine Learning to Estimate CLASS Positive Climate and Negative Climate](https://mlsanity.com/paper/2005.09525/) (2020)
- [Automated Evaluation of Classroom Instructional Support with LLMs and BoWs: Connecting Global Predictions to Specific Feedback](https://mlsanity.com/paper/2310.01132/) (2023)
- [Automated Assessment of Encouragement and Warmth in Classrooms Leveraging Multimodal Emotional Features and ChatGPT](https://mlsanity.com/paper/2404.15310/) (2024)
- [Can Open-Source LLMs Compete with Commercial Models? Exploring the Few-Shot Performance of Current GPT Models in Biomedical Tasks](https://mlsanity.com/paper/2407.13511/) (2024)
- [Measuring Five Accountable Talk Moves to Improve Instruction at Scale](https://mlsanity.com/paper/2311.10749/) (2023)
- [Analyzing Large Language Models for Classroom Discussion Assessment](https://mlsanity.com/paper/2406.08680/) (2024)

## Built on by (most cited first, 2 of 2)

- [Exploring Automated Recognition of Instructional Activity and Discourse from Multimodal Classroom Data](https://mlsanity.com/paper/2512.00087/) (2025)
- [From Scoring to Explanations: Evaluating SHAP and LLM Rationales for Rubric-based Teaching Quality Assessment](https://mlsanity.com/paper/2606.05180/) (2026)

## Abstract (arXiv)

Classroom discourse is an essential vehicle through which teaching and learning take place. Assessing different characteristics of discursive practices and linking them to student learning achievement enhances the understanding of teaching quality. Traditional assessments rely on manual coding of classroom observation protocols, which is time-consuming and costly. Despite many studies utilizing AI techniques to analyze classroom discourse at the utterance level, investigations into the evaluation of discursive practices throughout an entire lesson segment remain limited. To address this gap, our study proposes a novel text-centered multimodal fusion architecture to assess the quality of three discourse components grounded in the Global Teaching InSights (GTI) observation protocol: Nature of Discourse, Questioning, and Explanations. First, we employ attention mechanisms to capture inter- and intra-modal interactions from transcript, audio, and video streams. Second, a multi-task learning approach is adopted to jointly predict the quality scores of the three components. Third, we formulate the task as an ordinal classification problem to account for rating level order. The effectiveness of these designed elements is demonstrated through an ablation study on the GTI Germany dataset containing 92 videotaped math lessons. Our results highlight the dominant role of text modality in approaching this task. Integrating acoustic features enhances the model's consistency with human ratings, achieving an overall Quadratic Weighted Kappa score of 0.384, comparable to human inter-rater reliability (0.326). Our study lays the groundwork for the future development of automated discourse quality assessment to support teacher professional development through timely feedback on multidimensional discourse practices.