# Spatial-Temporal Recurrent Neural Network for Emotion Recognition

arXiv: https://arxiv.org/abs/1705.04515 · HTML: https://mlsanity.com/paper/1705.04515/

## Key facts

- Cited by 38 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 12 May 2017.

## Built on by (most cited first, 30 of 38)

- [Transformer-based Spatial-Temporal Feature Learning for EEG Decoding](https://mlsanity.com/paper/2106.11170/) (2021)
- [TSception: Capturing Temporal Dynamics and Spatial Asymmetry from EEG for Emotion Recognition](https://mlsanity.com/paper/2104.02935/) (2021)
- [EEG based Emotion Recognition: A Tutorial and Review](https://mlsanity.com/paper/2203.11279/) (2022)
- [Intensity-Aware Loss for Dynamic Facial Expression Recognition in the Wild](https://mlsanity.com/paper/2208.10335/) (2022)
- [CLIPER: A Unified Vision-Language Framework for In-the-Wild Facial Expression Recognition](https://mlsanity.com/paper/2303.00193/) (2023)
- [Learning from Heterogeneity: Generalizing Dynamic Facial Expression Recognition via Distributionally Robust Optimization](https://mlsanity.com/paper/2507.15765/) (2025)
- [GANSER: A Self-supervised Data Augmentation Framework for EEG-based Emotion Recognition](https://mlsanity.com/paper/2109.03124/) (2021)
- [HetEmotionNet: Two-Stream Heterogeneous Graph Recurrent Neural Network for Multi-modal Emotion Recognition](https://mlsanity.com/paper/2108.03354/) (2021)
- [GMSS: Graph-Based Multi-Task Self-Supervised Learning for EEG Emotion Recognition](https://mlsanity.com/paper/2205.01030/) (2022)
- [Multi-Modal Deep Learning for Credit Rating Prediction Using Text and Numerical Data Streams](https://mlsanity.com/paper/2304.10740/) (2023)
- [PARSE: Pairwise Alignment of Representations in Semi-Supervised EEG Learning for Emotion Recognition](https://mlsanity.com/paper/2202.05400/) (2022)
- [EEG-ITNet: An Explainable Inception Temporal Convolutional Network for Motor Imagery Classification](https://mlsanity.com/paper/2204.06947/) (2022)
- [Disentangling Identity and Pose for Facial Expression Recognition](https://mlsanity.com/paper/2208.08106/) (2022)
- [Spiking-Fer: Spiking Neural Network for Facial Expression Recognition With Event Cameras](https://mlsanity.com/paper/2304.10211/) (2023)
- [MASA-TCN: Multi-anchor Space-aware Temporal Convolutional Neural Networks for Continuous and Discrete EEG Emotion Recognition](https://mlsanity.com/paper/2308.16207/) (2023)
- [Distilling EEG Representations via Capsules for Affective Computing](https://mlsanity.com/paper/2105.00104/) (2021)
- [Virtual Temporal Samples for Recurrent Neural Networks: applied to semantic segmentation in agriculture](https://mlsanity.com/paper/2106.10118/) (2021)
- [Deep Recurrent Semi-Supervised EEG Representation Learning for Emotion Recognition](https://mlsanity.com/paper/2107.13505/) (2021)
- [Information Symmetry Matters: A Modal-Alternating Propagation Network for Few-Shot Learning](https://mlsanity.com/paper/2109.01295/) (2021)
- [Holistic Semi-Supervised Approaches for EEG Representation Learning](https://mlsanity.com/paper/2109.11732/) (2021)
- [A multi-level interpretable sleep stage scoring system by infusing experts' knowledge into a deep network architecture](https://mlsanity.com/paper/2207.04585/) (2022)
- [Progressive Graph Convolution Network for EEG Emotion Recognition](https://mlsanity.com/paper/2112.09069/) (2021)
- [Graph Convolutional Network with Connectivity Uncertainty for EEG-based Emotion Recognition](https://mlsanity.com/paper/2310.14165/) (2023)
- [Topological EEG Nonlinear Dynamics Analysis for Emotion Recognition](https://mlsanity.com/paper/2203.06895/) (2022)
- [Cross Task Neural Architecture Search for EEG Signal Classifications](https://mlsanity.com/paper/2210.06298/) (2022)
- [NSSI-Net: A Multi-Concept GAN for Non-Suicidal Self-Injury Detection Using High-Dimensional EEG in a Semi-Supervised Framework](https://mlsanity.com/paper/2410.12159/) (2024)
- [CMCRD: Cross-Modal Contrastive Representation Distillation for Emotion Recognition](https://mlsanity.com/paper/2504.09221/) (2025)
- [Multi-level Adversarial Spatio-temporal Learning for Footstep Pressure based FoG Detection](https://mlsanity.com/paper/2209.10770/) (2022)
- [Unsupervised Pairwise Learning Optimization Framework for Cross-Corpus EEG-Based Emotion Recognition Based on Prototype Representation](https://mlsanity.com/paper/2508.11663/) (2025)
- [Cross-domain EEG-based Emotion Recognition with Contrastive Learning](https://mlsanity.com/paper/2511.05293/) (2025)

## Abstract (arXiv)

Emotion analysis is a crucial problem to endow artifact machines with real intelligence in many large potential applications. As external appearances of human emotions, electroencephalogram (EEG) signals and video face signals are widely used to track and analyze human's affective information. According to their common characteristics of spatial-temporal volumes, in this paper we propose a novel deep learning framework named spatial-temporal recurrent neural network (STRNN) to unify the learning of two different signal sources into a spatial-temporal dependency model. In STRNN, to capture those spatially cooccurrent variations of human emotions, a multi-directional recurrent neural network (RNN) layer is employed to capture longrange contextual cues by traversing the spatial region of each time slice from multiple angles. Then a bi-directional temporal RNN layer is further used to learn discriminative temporal dependencies from the sequences concatenating spatial features of each time slice produced from the spatial RNN layer. To further select those salient regions of emotion representation, we impose sparse projection onto those hidden states of spatial and temporal domains, which actually also increases the model discriminant ability because of this global consideration. Consequently, such a two-layer RNN model builds spatial dependencies as well as temporal dependencies of the input signals. Experimental results on the public emotion datasets of EEG and facial expression demonstrate the proposed STRNN method is more competitive over those state-of-the-art methods.