# Distilling EEG Representations via Capsules for Affective Computing

arXiv: https://arxiv.org/abs/2105.00104 · HTML: https://mlsanity.com/paper/2105.00104/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 30 references that are in the map.
- First posted on arXiv 30 Apr 2021.

## Builds on (most cited first, 17 of 17)

- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [Layer Normalization](https://mlsanity.com/paper/1607.06450/) (2016)
- [Dynamic Routing Between Capsules](https://mlsanity.com/paper/1710.09829/) (2017)
- [Similarity-Preserving Knowledge Distillation](https://mlsanity.com/paper/1907.09682/) (2019)
- [Patient Knowledge Distillation for BERT Model Compression](https://mlsanity.com/paper/1908.09355/) (2019)
- [Unifying distillation and privileged information](https://mlsanity.com/paper/1511.03643/) (2015)
- [Regularizing Class-wise Predictions via Self-knowledge Distillation](https://mlsanity.com/paper/2003.13964/) (2020)
- [Spatio-Temporal Graph for Video Captioning with Knowledge Distillation](https://mlsanity.com/paper/2003.13942/) (2020)
- [EEG-Based Emotion Recognition Using Regularized Graph Neural Networks](https://mlsanity.com/paper/1907.07835/) (2019)
- [Joint Slot Filling and Intent Detection via Capsule Neural Networks](https://mlsanity.com/paper/1812.09471/) (2018)
- [Spatial-Temporal Recurrent Neural Network for Emotion Recognition](https://mlsanity.com/paper/1705.04515/) (2017)
- [COVID-CAPS: A Capsule Network-based Framework for Identification of COVID-19 cases from X-ray Images](https://mlsanity.com/paper/2004.02696/) (2020)
- [A Multimodal Approach to Estimating Vigilance Using EEG and Forehead EOG](https://mlsanity.com/paper/1611.08492/) (2016)
- [Dual Directed Capsule Network for Very Low Resolution Image Recognition](https://mlsanity.com/paper/1908.10027/) (2019)
- [Capsule Attention for Multimodal EEG-EOG Representation Learning with Application to Driver Vigilance Estimation](https://mlsanity.com/paper/1912.07812/) (2019)
- [CapsField: Light Field-based Face and Expression Recognition in the Wild using Capsule Routing](https://mlsanity.com/paper/2101.03503/) (2021)

## Built on by (most cited first, 8 of 8)

- [EEG based Emotion Recognition: A Tutorial and Review](https://mlsanity.com/paper/2203.11279/) (2022)
- [PARSE: Pairwise Alignment of Representations in Semi-Supervised EEG Learning for Emotion Recognition](https://mlsanity.com/paper/2202.05400/) (2022)
- [Partial Label Learning for Emotion Recognition from EEG](https://mlsanity.com/paper/2302.13170/) (2023)
- [Privileged Contrastive Pretraining for Multimodal Affect Modelling](https://mlsanity.com/paper/2508.03729/) (2025)
- [Enhancing Low-Density EEG-Based Brain-Computer Interfaces with Similarity-Keeping Knowledge Distillation](https://mlsanity.com/paper/2212.03329/) (2022)
- [Learning Oculomotor Behaviors from Scanpath](https://mlsanity.com/paper/2108.05025/) (2021)
- [Human Behavior-based Personalized Meal Recommendation and Menu Planning Social System](https://mlsanity.com/paper/2308.06549/) (2023)
- [Multi-Domain EEG Representation Learning with Orthogonal Mapping and Attention-based Fusion for Cognitive Load Classification](https://mlsanity.com/paper/2511.12394/) (2025)

## Abstract (arXiv)

Affective computing with Electroencephalogram (EEG) is a challenging task that requires cumbersome models to effectively learn the information contained in large-scale EEG signals, causing difficulties for real-time smart-device deployment. In this paper, we propose a novel knowledge distillation pipeline to distill EEG representations via capsule-based architectures for both classification and regression tasks. Our goal is to distill information from a heavy model to a lightweight model for subject-specific tasks. To this end, we first pre-train a large model (teacher network) on large number of training samples. Then, we employ the teacher network to learn the discriminative features embedded in capsules by adopting a lightweight model (student network) to mimic the teacher using the privileged knowledge. Such privileged information learned by the teacher contain similarities among capsules and are only available during the training stage of the student network. We evaluate the proposed architecture on two large-scale public EEG datasets, showing that our framework consistently enables student networks with different compression ratios to effectively learn from the teacher, even when provided with limited training samples. Lastly, our method achieves state-of-the-art results on one of the two datasets.