# Automated Assessment of Encouragement and Warmth in Classrooms Leveraging Multimodal Emotional Features and ChatGPT

arXiv: https://arxiv.org/abs/2404.15310 · HTML: https://mlsanity.com/paper/2404.15310/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 12 of its 32 references that are in the map.
- First posted on arXiv 1 Apr 2024.

## Builds on (most cited first, 12 of 12)

- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [A Unified Approach to Interpreting Model Predictions](https://mlsanity.com/paper/1705.07874/) (2017)
- [wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations](https://mlsanity.com/paper/2006.11477/) (2020)
- [ChatGPT Outperforms Crowd-Workers for Text-Annotation Tasks](https://mlsanity.com/paper/2303.15056/) (2023)
- [Unsupervised Cross-lingual Representation Learning for Speech Recognition](https://mlsanity.com/paper/2006.13979/) (2020)
- [AffectNet: A Database for Facial Expression, Valence, and Arousal Computing in the Wild](https://mlsanity.com/paper/1708.03985/) (2017)
- [Emotion Recognition from Speech Using Wav2vec 2.0 Embeddings](https://mlsanity.com/paper/2104.03502/) (2021)
- [Is ChatGPT a Good Teacher Coach? Measuring Zero-Shot Performance For Scoring and Providing Actionable Insights on Classroom Instruction](https://mlsanity.com/paper/2306.03090/) (2023)
- [Multimodal Engagement Analysis from Facial Videos in the Classroom](https://mlsanity.com/paper/2101.04215/) (2021)
- [Evaluating the Explainers: Black-Box Explainable Machine Learning for Student Success Prediction in MOOCs](https://mlsanity.com/paper/2207.00551/) (2022)
- [Toward Automated Classroom Observation: Multimodal Machine Learning to Estimate CLASS Positive Climate and Negative Climate](https://mlsanity.com/paper/2005.09525/) (2020)
- [Automated Evaluation of Classroom Instructional Support with LLMs and BoWs: Connecting Global Predictions to Specific Feedback](https://mlsanity.com/paper/2310.01132/) (2023)

## Built on by (most cited first, 7 of 7)

- [CUIfy the XR: An Open-Source Package to Embed LLM-powered Conversational Agents in XR](https://mlsanity.com/paper/2411.04671/) (2024)
- [From Passive Watching to Active Learning: Empowering Proactive Participation in Digital Classrooms with AI Video Assistant](https://mlsanity.com/paper/2409.15843/) (2024)
- [Exploring Context-aware and LLM-driven Locomotion for Immersive Virtual Reality](https://mlsanity.com/paper/2504.17331/) (2025)
- [Multimodal Assessment of Classroom Discourse Quality: A Text-Centered Attention-Based Multi-Task Learning Approach](https://mlsanity.com/paper/2505.07902/) (2025)
- [TACLA: An LLM-Based Multi-Agent Tool for Transactional Analysis Training in Education](https://mlsanity.com/paper/2510.17913/) (2025)
- [Exploring Automated Recognition of Instructional Activity and Discourse from Multimodal Classroom Data](https://mlsanity.com/paper/2512.00087/) (2025)
- [From Scoring to Explanations: Evaluating SHAP and LLM Rationales for Rubric-based Teaching Quality Assessment](https://mlsanity.com/paper/2606.05180/) (2026)

## Abstract (arXiv)

Classroom observation protocols standardize the assessment of teaching effectiveness and facilitate comprehension of classroom interactions. Whereas these protocols offer teachers specific feedback on their teaching practices, the manual coding by human raters is resource-intensive and often unreliable. This has sparked interest in developing AI-driven, cost-effective methods for automating such holistic coding. Our work explores a multimodal approach to automatically estimating encouragement and warmth in classrooms, a key component of the Global Teaching Insights (GTI) study's observation protocol. To this end, we employed facial and speech emotion recognition with sentiment analysis to extract interpretable features from video, audio, and transcript data. The prediction task involved both classification and regression methods. Additionally, in light of recent large language models' remarkable text annotation capabilities, we evaluated ChatGPT's zero-shot performance on this scoring task based on transcripts. We demonstrated our approach on the GTI dataset, comprising 367 16-minute video segments from 92 authentic lesson recordings. The inferences of GPT-4 and the best-trained model yielded correlations of r = .341 and r = .441 with human ratings, respectively. Combining estimates from both models through averaging, an ensemble approach achieved a correlation of r = .513, comparable to human inter-rater reliability. Our model explanation analysis indicated that text sentiment features were the primary contributors to the trained model's decisions. Moreover, GPT-4 could deliver logical and concrete reasoning as potential teacher guidelines. Our findings provide insights into using advanced, multimodal techniques for automated classroom observation, aiming to foster teacher training through frequent and valuable feedback.