# Improving state-of-the-art in Detecting Student Engagement with Resnet and TCN Hybrid Network

arXiv: https://arxiv.org/abs/2104.10122 · HTML: https://mlsanity.com/paper/2104.10122/

## Key facts

- Cited by 22 papers in the map (data through 14 Oct 2026).
- Builds on 9 of its 30 references that are in the map.
- First posted on arXiv 20 Apr 2021.

## Builds on (most cited first, 9 of 9)

- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Going Deeper with Convolutions](https://mlsanity.com/paper/1409.4842/) (2014)
- [An Empirical Evaluation of Generic Convolutional and Recurrent Networks for Sequence Modeling](https://mlsanity.com/paper/1803.01271/) (2018)
- [Long-term Recurrent Convolutional Networks for Visual Recognition and Description](https://mlsanity.com/paper/1411.4389/) (2014)
- [Learning To Score Olympic Events](https://mlsanity.com/paper/1611.05125/) (2016)
- [End-to-end Audiovisual Speech Recognition](https://mlsanity.com/paper/1802.06424/) (2018)
- [DAiSEE: Towards User Engagement Recognition in the Wild](https://mlsanity.com/paper/1609.01885/) (2016)
- [Multimodal Engagement Analysis from Facial Videos in the Classroom](https://mlsanity.com/paper/2101.04215/) (2021)
- [Lip-reading with Densely Connected Temporal Convolutional Networks](https://mlsanity.com/paper/2009.14233/) (2020)

## Built on by (most cited first, 22 of 22)

- [Affect-driven Ordinal Engagement Measurement from Video](https://mlsanity.com/paper/2106.10882/) (2021)
- [Class-attention Video Transformer for Engagement Intensity Prediction](https://mlsanity.com/paper/2208.07216/) (2022)
- [Engagement Measurement Based on Facial Landmarks and Spatial-Temporal Graph Convolutional Networks](https://mlsanity.com/paper/2403.17175/) (2024)
- [TCCT-Net: Two-Stream Network Architecture for Fast and Efficient Engagement Estimation via Behavioral Feature Signals](https://mlsanity.com/paper/2404.09474/) (2024)
- [A Decade of Action Quality Assessment: Largest Systematic Survey of Trends, Challenges, and Future Directions](https://mlsanity.com/paper/2502.02817/) (2025)
- [Detecting Disengagement in Virtual Learning as an Anomaly using Temporal Convolutional Network Autoencoder](https://mlsanity.com/paper/2211.06870/) (2022)
- [Bag of States: A Non-sequential Approach to Video-based Engagement Measurement](https://mlsanity.com/paper/2301.06730/) (2023)
- [Analyzing Participants' Engagement during Online Meetings Using Unsupervised Remote Photoplethysmography with Behavioral Features](https://mlsanity.com/paper/2404.04394/) (2024)
- [Computational Analysis of Stress, Depression and Engagement in Mental Health: A Survey](https://mlsanity.com/paper/2403.08824/) (2024)
- [A General Model for Detecting Learner Engagement: Implementation and Evaluation](https://mlsanity.com/paper/2405.04251/) (2024)
- [CMOSE: Comprehensive Multi-Modality Online Student Engagement Dataset with High-Quality Labels](https://mlsanity.com/paper/2312.09066/) (2023)
- [Deep Spectro-temporal Artifacts for Detecting Synthesized Speech](https://mlsanity.com/paper/2210.05254/) (2022)
- [ViBED-Net: Video Based Engagement Detection Network Using Face-Aware and Scene-Aware Spatiotemporal Cues](https://mlsanity.com/paper/2510.18016/) (2025)
- [Supervised Contrastive Learning for Ordinal Engagement Measurement](https://mlsanity.com/paper/2505.20676/) (2025)
- [PriorNet: Prior-Guided Engagement Estimation from Face Video](https://mlsanity.com/paper/2605.03615/) (2026)
- [Inconsistencies in the Definition and Annotation of Student Engagement in Virtual Learning Datasets: A Critical Review](https://mlsanity.com/paper/2208.04548/) (2022)
- [VisioPhysioENet: Visual Physiological Engagement Detection Network](https://mlsanity.com/paper/2409.16126/) (2024)
- [OPEN: A Benchmark Dataset and Baseline for Older Adult Patient Engagement Recognition in Virtual Rehabilitation Learning Environments](https://mlsanity.com/paper/2507.17959/) (2025)
- [Automatic Assessment of Students' Classroom Engagement with Bias Mitigated Multi-task Model](https://mlsanity.com/paper/2510.22057/) (2025)
- [ConfusionBench: An Expert-Validated Benchmark for Confusion Recognition and Localization in Educational Videos](https://mlsanity.com/paper/2603.17267/) (2026)
- [Zero-Shot Vision-Language Models for Classroom Engagement Recognition: A Benchmark Study of Prompt Sensitivity and Cross-Dataset Generalization](https://mlsanity.com/paper/2606.21861/) (2026)
- [Dimension-Specific Imbalance and an Adaptive Hybrid Label Strategy for Multi-Task Affective State Recognition in Classroom Video](https://mlsanity.com/paper/2609.31750/) (2026)

## Abstract (arXiv)

Automatic detection of students' engagement in online learning settings is a key element to improve the quality of learning and to deliver personalized learning materials to them. Varying levels of engagement exhibited by students in an online classroom is an affective behavior that takes place over space and time. Therefore, we formulate detecting levels of students' engagement from videos as a spatio-temporal classification problem. In this paper, we present a novel end-to-end Residual Network (ResNet) and Temporal Convolutional Network (TCN) hybrid neural network architecture for students' engagement level detection in videos. The 2D ResNet extracts spatial features from consecutive video frames, and the TCN analyzes the temporal changes in video frames to detect the level of engagement. The spatial and temporal arms of the hybrid network are jointly trained on raw video frames of a large publicly available students' engagement detection dataset, DAiSEE. We compared our method with several competing students' engagement detection methods on this dataset. The ResNet+TCN architecture outperforms all other studied methods, improves the state-of-the-art engagement level detection accuracy, and sets a new baseline for future research.