# Capsule Attention for Multimodal EEG-EOG Representation Learning with Application to Driver Vigilance Estimation

arXiv: https://arxiv.org/abs/1912.07812 · HTML: https://mlsanity.com/paper/1912.07812/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 17 Dec 2019.

## Built on by (most cited first, 9 of 9)

- [Multimodal Brain-Computer Interface for In-Vehicle Driver Cognitive Load Measurement: Dataset and Baselines](https://mlsanity.com/paper/2304.04273/) (2023)
- [PARSE: Pairwise Alignment of Representations in Semi-Supervised EEG Learning for Emotion Recognition](https://mlsanity.com/paper/2202.05400/) (2022)
- [Distilling EEG Representations via Capsules for Affective Computing](https://mlsanity.com/paper/2105.00104/) (2021)
- [AttX: Attentive Cross-Connections for Fusion of Wearable Signals in Emotion Recognition](https://mlsanity.com/paper/2206.04625/) (2022)
- [Holistic Semi-Supervised Approaches for EEG Representation Learning](https://mlsanity.com/paper/2109.11732/) (2021)
- [Partial Label Learning for Emotion Recognition from EEG](https://mlsanity.com/paper/2302.13170/) (2023)
- [DistilCLIP-EEG: Enhancing Epileptic Seizure Detection Through Multi-modal Learning and Knowledge Distillation](https://mlsanity.com/paper/2510.13497/) (2025)
- [An embedding for EEG signals learned using a triplet loss](https://mlsanity.com/paper/2304.06495/) (2023)
- [Understanding Mental States in Active and Autonomous Driving with EEG](https://mlsanity.com/paper/2512.09190/) (2025)

## Abstract (arXiv)

Driver vigilance estimation is an important task for transportation safety. Wearable and portable brain-computer interface devices provide a powerful means for real-time monitoring of the vigilance level of drivers to help with avoiding distracted or impaired driving. In this paper, we propose a novel multimodal architecture for in-vehicle vigilance estimation from Electroencephalogram and Electrooculogram. To enable the system to focus on the most salient parts of the learned multimodal representations, we propose an architecture composed of a capsule attention mechanism following a deep Long Short-Term Memory (LSTM) network. Our model learns hierarchical dependencies in the data through the LSTM and capsule feature representation layers. To better explore the discriminative ability of the learned representations, we study the effect of the proposed capsule attention mechanism including the number of dynamic routing iterations as well as other parameters. Experiments show the robustness of our method by outperforming other solutions and baseline techniques, setting a new state-of-the-art. We then provide an analysis on different frequency bands and brain regions to evaluate their suitability for driver vigilance estimation. Lastly, an analysis on the role of capsule attention, multimodality, and robustness to noise is performed, highlighting the advantages of our approach.