# Multimodal Engagement Analysis from Facial Videos in the Classroom

arXiv: https://arxiv.org/abs/2101.04215 · HTML: https://mlsanity.com/paper/2101.04215/

## Key facts

- Cited by 27 papers in the map (data through 14 Oct 2026).
- Builds on 14 of its 106 references that are in the map.
- First posted on arXiv 11 Jan 2021.

## Builds on (most cited first, 14 of 14)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Sequence to Sequence Learning with Neural Networks](https://mlsanity.com/paper/1409.3215/) (2014)
- [AffectNet: A Database for Facial Expression, Valence, and Arousal Computing in the Wild](https://mlsanity.com/paper/1708.03985/) (2017)
- [Long-term Recurrent Convolutional Networks for Visual Recognition and Description](https://mlsanity.com/paper/1411.4389/) (2014)
- [RetinaFace: Single-stage Dense Face Localisation in the Wild](https://mlsanity.com/paper/1905.00641/) (2019)
- [Fine-Grained Head Pose Estimation Without Keypoints](https://mlsanity.com/paper/1710.00925/) (2017)
- [Prediction and Localization of Student Engagement in the Wild](https://mlsanity.com/paper/1804.00858/) (2018)
- [Personalized Machine Learning for Robot Perception of Affect and Engagement in Autism Therapy](https://mlsanity.com/paper/1802.01186/) (2018)
- [Deep Learning for Human Affect Recognition: Insights and New Developments](https://mlsanity.com/paper/1901.02884/) (2019)
- [Affective Computing for Large-Scale Heterogeneous Multimedia Data: A Survey](https://mlsanity.com/paper/1911.05609/) (2019)
- [Deep learning for affective computing: text-based emotion recognition in decision support](https://mlsanity.com/paper/1803.06397/) (2018)
- [Multi-modal Active Learning From Human Data: A Deep Reinforcement Learning Approach](https://mlsanity.com/paper/1906.03098/) (2019)
- [EEG in the classroom: Synchronised neural recordings during video presentation](https://mlsanity.com/paper/1604.03019/) (2016)
- [Automated Anonymisation of Visual and Audio Data in Classroom Studies](https://mlsanity.com/paper/2001.05080/) (2020)

## Built on by (most cited first, 27 of 27)

- [Foundations and Trends in Multimodal Machine Learning: Principles, Challenges, and Open Questions](https://mlsanity.com/paper/2209.03430/) (2022)
- [Improving state-of-the-art in Detecting Student Engagement with Resnet and TCN Hybrid Network](https://mlsanity.com/paper/2104.10122/) (2021)
- [Affect-driven Ordinal Engagement Measurement from Video](https://mlsanity.com/paper/2106.10882/) (2021)
- [MC-ViViT: Multi-branch Classifier-ViViT to detect Mild Cognitive Impairment in older adults using facial videos](https://mlsanity.com/paper/2304.05292/) (2023)
- [Automated Assessment of Encouragement and Warmth in Classrooms Leveraging Multimodal Emotional Features and ChatGPT](https://mlsanity.com/paper/2404.15310/) (2024)
- [Detecting Disengagement in Virtual Learning as an Anomaly using Temporal Convolutional Network Autoencoder](https://mlsanity.com/paper/2211.06870/) (2022)
- [Bag of States: A Non-sequential Approach to Video-based Engagement Measurement](https://mlsanity.com/paper/2301.06730/) (2023)
- [Beyond One-Size-Fits-All: A Survey of Personalized Affective Computing in Human-Agent Interaction](https://mlsanity.com/paper/2304.00377/) (2023)
- [Computational Analysis of Stress, Depression and Engagement in Mental Health: A Survey](https://mlsanity.com/paper/2403.08824/) (2024)
- [Multimodal Methods for Analyzing Learning and Training Environments: A Systematic Literature Review](https://mlsanity.com/paper/2408.14491/) (2024)
- [CMOSE: Comprehensive Multi-Modality Online Student Engagement Dataset with High-Quality Labels](https://mlsanity.com/paper/2312.09066/) (2023)
- [Measuring Student Behavioral Engagement using Histogram of Actions](https://mlsanity.com/paper/2307.09420/) (2023)
- [Leveraging Multi-Modal Saliency and Fusion for Gaze Target Detection](https://mlsanity.com/paper/2504.19271/) (2025)
- [Automated Visual Attention Detection using Mobile Eye Tracking in Behavioral Classroom Studies](https://mlsanity.com/paper/2505.07552/) (2025)
- [Multimodal Assessment of Classroom Discourse Quality: A Text-Centered Attention-Based Multi-Task Learning Approach](https://mlsanity.com/paper/2505.07902/) (2025)
- [Deep Adaptation of Adult-Child Facial Expressions by Fusing Landmark Features](https://mlsanity.com/paper/2209.08614/) (2022)
- [AI-Based Facial Emotion Recognition Solutions for Education: A Study of Teacher-User and Other Categories](https://mlsanity.com/paper/2308.15119/) (2023)
- [Leveraging Language Models for Emotion and Behavior Analysis in Education](https://mlsanity.com/paper/2408.06874/) (2024)
- [Exploring Automated Recognition of Instructional Activity and Discourse from Multimodal Classroom Data](https://mlsanity.com/paper/2512.00087/) (2025)
- [Inconsistencies in the Definition and Annotation of Student Engagement in Virtual Learning Datasets: A Critical Review](https://mlsanity.com/paper/2208.04548/) (2022)
- [Multimodality in Online Education: A Comparative Study](https://mlsanity.com/paper/2312.05797/) (2023)
- [VisioPhysioENet: Visual Physiological Engagement Detection Network](https://mlsanity.com/paper/2409.16126/) (2024)
- [Research on Short-Video Platform User Decision-Making via Multimodal Temporal Modeling and Reinforcement Learning](https://mlsanity.com/paper/2509.12269/) (2025)
- [DS-HGCN: A Dual-Stream Hypergraph Convolutional Network for Predicting Student Engagement via Social Contagion](https://mlsanity.com/paper/2512.20059/) (2025)
- [Safeguarding Privacy: Privacy-Preserving Detection of Mind Wandering and Disengagement Using Federated Learning in Online Education](https://mlsanity.com/paper/2602.09904/) (2026)
- [Toward Scalable Co-located Practical Learning: Assisting with Computer Vision and Multimodal Analytics](https://mlsanity.com/paper/2603.13679/) (2026)
- [Cross-modal Affinity-aligned Multimodal Learning Analytics for Predicting Student Collaboration Satisfaction in Game-Based Learning](https://mlsanity.com/paper/2605.16806/) (2026)

## Abstract (arXiv)

Student engagement is a key construct for learning and teaching. While most of the literature explored the student engagement analysis on computer-based settings, this paper extends that focus to classroom instruction. To best examine student visual engagement in the classroom, we conducted a study utilizing the audiovisual recordings of classes at a secondary school over one and a half month's time, acquired continuous engagement labeling per student (N=15) in repeated sessions, and explored computer vision methods to classify engagement levels from faces in the classroom. We trained deep embeddings for attentional and emotional features, training Attention-Net for head pose estimation and Affect-Net for facial expression recognition. We additionally trained different engagement classifiers, consisting of Support Vector Machines, Random Forest, Multilayer Perceptron, and Long Short-Term Memory, for both features. The best performing engagement classifiers achieved AUCs of .620 and .720 in Grades 8 and 12, respectively. We further investigated fusion strategies and found score-level fusion either improves the engagement classifiers or is on par with the best performing modality. We also investigated the effect of personalization and found that using only 60-seconds of person-specific data selected by margin uncertainty of the base classifier yielded an average AUC improvement of .084. 4.Our main aim with this work is to provide the technical means to facilitate the manual data analysis of classroom videos in research on teaching quality and in the context of teacher training.