# VisioPhysioENet: Visual Physiological Engagement Detection Network

arXiv: https://arxiv.org/abs/2409.16126 · HTML: https://mlsanity.com/paper/2409.16126/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 12 of its 47 references that are in the map.
- First posted on arXiv 24 Sep 2024.

## Builds on (most cited first, 12 of 12)

- [Self-supervised ECG Representation Learning for Emotion Recognition](https://mlsanity.com/paper/2002.03898/) (2020)
- [DAiSEE: Towards User Engagement Recognition in the Wild](https://mlsanity.com/paper/1609.01885/) (2016)
- [Multimodal Engagement Analysis from Facial Videos in the Classroom](https://mlsanity.com/paper/2101.04215/) (2021)
- [Improving state-of-the-art in Detecting Student Engagement with Resnet and TCN Hybrid Network](https://mlsanity.com/paper/2104.10122/) (2021)
- [Do I Have Your Attention: A Large Scale Engagement Prediction Dataset and Baselines](https://mlsanity.com/paper/2302.00431/) (2023)
- [Affect-driven Ordinal Engagement Measurement from Video](https://mlsanity.com/paper/2106.10882/) (2021)
- [Student Engagement Detection Using Emotion Analysis, Eye Tracking and Head Movement with Machine Learning](https://mlsanity.com/paper/1909.12913/) (2019)
- [TCCT-Net: Two-Stream Network Architecture for Fast and Efficient Engagement Estimation via Behavioral Feature Signals](https://mlsanity.com/paper/2404.09474/) (2024)
- [Analyzing Participants' Engagement during Online Meetings Using Unsupervised Remote Photoplethysmography with Behavioral Features](https://mlsanity.com/paper/2404.04394/) (2024)
- [VISTANet: VIsual Spoken Textual Additive Net for Interpretable Multimodal Emotion Recognition](https://mlsanity.com/paper/2208.11450/) (2022)
- [Interpretable Multimodal Emotion Recognition using Facial Features and Physiological Signals](https://mlsanity.com/paper/2306.02845/) (2023)
- [Analysis of face detection, face landmarking, and face recognition performance with masked face images](https://mlsanity.com/paper/2207.06478/) (2022)

## Abstract (arXiv)

This paper presents VisioPhysioENet, a novel multimodal system that leverages visual and physiological signals to detect learner engagement. It employs a two-level approach for extracting both visual and physiological features. For visual feature extraction, Dlib is used to detect facial landmarks, while OpenCV provides additional estimations. The face recognition library, built on Dlib, is used to identify the facial region of interest specifically for physiological signal extraction. Physiological signals are then extracted using the plane-orthogonal-toskin method to assess cardiovascular activity. These features are integrated using advanced machine learning classifiers, enhancing the detection of various levels of engagement. We thoroughly tested VisioPhysioENet on the DAiSEE dataset. It achieved an accuracy of 63.09%. This shows it can better identify different levels of engagement compared to many existing methods. It performed 8.6% better than the only other model that uses both physiological and visual features.