# Engagement Measurement Based on Facial Landmarks and Spatial-Temporal Graph Convolutional Networks

arXiv: https://arxiv.org/abs/2403.17175 · HTML: https://mlsanity.com/paper/2403.17175/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 16 of its 45 references that are in the map.
- First posted on arXiv 25 Mar 2024.

## Builds on (most cited first, 16 of 16)

- [Spatial Temporal Graph Convolutional Networks for Skeleton-Based Action Recognition](https://mlsanity.com/paper/1801.07455/) (2018)
- [MediaPipe: A Framework for Building Perception Pipelines](https://mlsanity.com/paper/1906.08172/) (2019)
- [MARLIN: Masked Autoencoder for facial video Representation LearnINg](https://mlsanity.com/paper/2211.06627/) (2022)
- [Rendering of Eyes for Eye-Shape Registration and Gaze Estimation](https://mlsanity.com/paper/1505.05916/) (2015)
- [DAiSEE: Towards User Engagement Recognition in the Wild](https://mlsanity.com/paper/1609.01885/) (2016)
- [Attention Mesh: High-fidelity Face Mesh Prediction in Real-time](https://mlsanity.com/paper/2006.10962/) (2020)
- [Improving state-of-the-art in Detecting Student Engagement with Resnet and TCN Hybrid Network](https://mlsanity.com/paper/2104.10122/) (2021)
- [Do I Have Your Attention: A Large Scale Engagement Prediction Dataset and Baselines](https://mlsanity.com/paper/2302.00431/) (2023)
- [Graph-based Facial Affect Analysis: A Review](https://mlsanity.com/paper/2103.15599/) (2021)
- [Affect-driven Ordinal Engagement Measurement from Video](https://mlsanity.com/paper/2106.10882/) (2021)
- [Engagement Detection with Multi-Task Training in E-Learning Environments](https://mlsanity.com/paper/2204.04020/) (2022)
- [Class-attention Video Transformer for Engagement Intensity Prediction](https://mlsanity.com/paper/2208.07216/) (2022)
- [Geometric Graph Representation with Learnable Graph Structure and Adaptive AU Constraint for Micro-Expression Recognition](https://mlsanity.com/paper/2205.00380/) (2022)
- [Detecting Disengagement in Virtual Learning as an Anomaly using Temporal Convolutional Network Autoencoder](https://mlsanity.com/paper/2211.06870/) (2022)
- [Bag of States: A Non-sequential Approach to Video-based Engagement Measurement](https://mlsanity.com/paper/2301.06730/) (2023)
- [Cross-Modal Video to Body-joints Augmentation for Rehabilitation Exercise Quality Assessment](https://mlsanity.com/paper/2306.09546/) (2023)

## Built on by (most cited first, 7 of 7)

- [Explainable graph attention network for stress recognition (StressGAT) via differential action units](https://mlsanity.com/paper/2607.20819/) (2026)
- [Supervised Contrastive Learning for Ordinal Engagement Measurement](https://mlsanity.com/paper/2505.20676/) (2025)
- [FairSSL: Fair Multimodal Self-Supervised Learning](https://mlsanity.com/paper/2508.16748/) (2025)
- [PriorNet: Prior-Guided Engagement Estimation from Face Video](https://mlsanity.com/paper/2605.03615/) (2026)
- [OPEN: A Benchmark Dataset and Baseline for Older Adult Patient Engagement Recognition in Virtual Rehabilitation Learning Environments](https://mlsanity.com/paper/2507.17959/) (2025)
- [Facial Expression Recognition with YOLOv11 and YOLOv12: A Comparative Study](https://mlsanity.com/paper/2511.10940/) (2025)
- [Variational Encoder--Multi-Decoder (VE-MD) for Privacy-by-functional-design (Group) Emotion Recognition](https://mlsanity.com/paper/2604.02397/) (2026)

## Abstract (arXiv)

Engagement in virtual learning is crucial for a variety of factors including student satisfaction, performance, and compliance with learning programs, but measuring it is a challenging task. There is therefore considerable interest in utilizing artificial intelligence and affective computing to measure engagement in natural settings as well as on a large scale. This paper introduces a novel, privacy-preserving method for engagement measurement from videos. It uses facial landmarks, which carry no personally identifiable information, extracted from videos via the MediaPipe deep learning solution. The extracted facial landmarks are fed to Spatial-Temporal Graph Convolutional Networks (ST-GCNs) to output the engagement level of the student in the video. To integrate the ordinal nature of the engagement variable into the training process, ST-GCNs undergo training in a novel ordinal learning framework based on transfer learning. Experimental results on two video student engagement measurement datasets show the superiority of the proposed method compared to previous methods with improved state-of-the-art on the EngageNet dataset with a 3.1% improvement in four-class engagement level classification accuracy and on the Online Student Engagement dataset with a 1.5% improvement in binary engagement classification accuracy. Gradient-weighted Class Activation Mapping (Grad-CAM) was applied to the developed ST-GCNs to interpret the engagement measurements obtained by the proposed method in both the spatial and temporal domains. The relatively lightweight and fast ST-GCN and its integration with the real-time MediaPipe make the proposed approach capable of being deployed on virtual learning platforms and measuring engagement in real-time.