# Bag of States: A Non-sequential Approach to Video-based Engagement Measurement

arXiv: https://arxiv.org/abs/2301.06730 · HTML: https://mlsanity.com/paper/2301.06730/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 73 references that are in the map.
- First posted on arXiv 17 Jan 2023.

## Builds on (most cited first, 17 of 17)

- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [An Empirical Evaluation of Generic Convolutional and Recurrent Networks for Sequence Modeling](https://mlsanity.com/paper/1803.01271/) (2018)
- [Realtime Multi-Person 2D Pose Estimation using Part Affinity Fields](https://mlsanity.com/paper/1611.08050/) (2016)
- [MediaPipe: A Framework for Building Perception Pipelines](https://mlsanity.com/paper/1906.08172/) (2019)
- [AffectNet: A Database for Facial Expression, Valence, and Arousal Computing in the Wild](https://mlsanity.com/paper/1708.03985/) (2017)
- [Fundamentals of Recurrent Neural Network (RNN) and Long Short-Term Memory (LSTM) Network](https://mlsanity.com/paper/1808.03314/) (2018)
- [DAiSEE: Towards User Engagement Recognition in the Wild](https://mlsanity.com/paper/1609.01885/) (2016)
- [Prediction and Localization of Student Engagement in the Wild](https://mlsanity.com/paper/1804.00858/) (2018)
- [Multimodal Engagement Analysis from Facial Videos in the Classroom](https://mlsanity.com/paper/2101.04215/) (2021)
- [Improving state-of-the-art in Detecting Student Engagement with Resnet and TCN Hybrid Network](https://mlsanity.com/paper/2104.10122/) (2021)
- [MIMAMO Net: Integrating Micro- and Macro-motion for Video Emotion Recognition](https://mlsanity.com/paper/1911.09784/) (2019)
- [Bag-of-Words vs. Graph vs. Sequence in Text Classification: Questioning the Necessity of Text-Graphs and the Surprising Strength of a Wide MLP](https://mlsanity.com/paper/2109.03777/) (2021)
- [Affect-driven Ordinal Engagement Measurement from Video](https://mlsanity.com/paper/2106.10882/) (2021)
- [Engagement Detection with Multi-Task Training in E-Learning Environments](https://mlsanity.com/paper/2204.04020/) (2022)
- [Class-attention Video Transformer for Engagement Intensity Prediction](https://mlsanity.com/paper/2208.07216/) (2022)
- [Augmenting Bag-of-Words: Data-Driven Discovery of Temporal and Structural Information for Activity Recognition](https://mlsanity.com/paper/1510.02071/) (2015)
- [Detecting Disengagement in Virtual Learning as an Anomaly using Temporal Convolutional Network Autoencoder](https://mlsanity.com/paper/2211.06870/) (2022)

## Built on by (most cited first, 5 of 5)

- [Engagement Measurement Based on Facial Landmarks and Spatial-Temporal Graph Convolutional Networks](https://mlsanity.com/paper/2403.17175/) (2024)
- [A General Model for Detecting Learner Engagement: Implementation and Evaluation](https://mlsanity.com/paper/2405.04251/) (2024)
- [Multimodal Sensor Dataset for Monitoring Older Adults Post Lower-Limb Fractures in Community Settings](https://mlsanity.com/paper/2501.13888/) (2025)
- [Supervised Contrastive Learning for Ordinal Engagement Measurement](https://mlsanity.com/paper/2505.20676/) (2025)
- [OPEN: A Benchmark Dataset and Baseline for Older Adult Patient Engagement Recognition in Virtual Rehabilitation Learning Environments](https://mlsanity.com/paper/2507.17959/) (2025)

## Abstract (arXiv)

Automatic measurement of student engagement provides helpful information for instructors to meet learning program objectives and individualize program delivery. Students' behavioral and emotional states need to be analyzed at fine-grained time scales in order to measure their level of engagement. Many existing approaches have developed sequential and spatiotemporal models, such as recurrent neural networks, temporal convolutional networks, and three-dimensional convolutional neural networks, for measuring student engagement from videos. These models are trained to incorporate the order of behavioral and emotional states of students into video analysis and output their level of engagement. In this paper, backed by educational psychology, we question the necessity of modeling the order of behavioral and emotional states of students in measuring their engagement. We develop bag-of-words-based models in which only the occurrence of behavioral and emotional states of students is modeled and analyzed and not the order in which they occur. Behavioral and affective features are extracted from videos and analyzed by the proposed models to determine the level of engagement in an ordinal-output classification setting. Compared to the existing sequential and spatiotemporal approaches for engagement measurement, the proposed non-sequential approach improves the state-of-the-art results. According to experimental results, our method significantly improved engagement level classification accuracy on the IIITB Online SE dataset by 26% compared to sequential models and achieved engagement level classification accuracy as high as 66.58% on the DAiSEE student engagement dataset.