# An EEG-Based Multi-Modal Emotion Database with Both Posed and Authentic Facial Actions for Emotion Analysis

arXiv: https://arxiv.org/abs/2203.15829 · HTML: https://mlsanity.com/paper/2203.15829/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 4 of its 36 references that are in the map.
- First posted on arXiv 29 Mar 2022.

## Builds on (most cited first, 4 of 4)

- [Domain-Adversarial Training of Neural Networks](https://mlsanity.com/paper/1505.07818/) (2015)
- [DeCAF: A Deep Convolutional Activation Feature for Generic Visual Recognition](https://mlsanity.com/paper/1310.1531/) (2013)
- [Survey on RGB, 3D, Thermal, and Multimodal Approaches for Facial Expression Recognition: History, Trends, and Affect-related Applications](https://mlsanity.com/paper/1606.03237/) (2016)
- [Peak-Piloted Deep Network for Facial Expression Recognition](https://mlsanity.com/paper/1607.06997/) (2016)

## Built on by (most cited first, 6 of 6)

- [A Survey on Physiological Signal Based Emotion Recognition](https://mlsanity.com/paper/2205.10466/) (2022)
- [Multimodal Channel-Mixing: Channel and Spatial Masked AutoEncoder on Facial Action Unit Detection](https://mlsanity.com/paper/2209.12244/) (2022)
- [Mind-to-Face: Neural-Driven Photorealistic Avatar Synthesis via EEG Decoding](https://mlsanity.com/paper/2512.04313/) (2025)
- [Unifying EEG and Speech for Emotion Recognition: A Two-Step Joint Learning Framework for Handling Missing EEG Data During Inference](https://mlsanity.com/paper/2503.18964/) (2025)
- [MindAU: EEG-Conditioned Facial Action Unit Editing via Dual-Stream Manifold Alignment](https://mlsanity.com/paper/2607.00410/) (2026)
- [A Comprehensive Review of Multimodal Facial State Analysis: Tasks, Methods, and Resources](https://mlsanity.com/paper/2609.13255/) (2026)

## Abstract (arXiv)

Emotion is an experience associated with a particular pattern of physiological activity along with different physiological, behavioral and cognitive changes. One behavioral change is facial expression, which has been studied extensively over the past few decades. Facial behavior varies with a person's emotion according to differences in terms of culture, personality, age, context, and environment. In recent years, physiological activities have been used to study emotional responses. A typical signal is the electroencephalogram (EEG), which measures brain activity. Most of existing EEG-based emotion analysis has overlooked the role of facial expression changes. There exits little research on the relationship between facial behavior and brain signals due to the lack of dataset measuring both EEG and facial action signals simultaneously. To address this problem, we propose to develop a new database by collecting facial expressions, action units, and EEGs simultaneously. We recorded the EEGs and face videos of both posed facial actions and spontaneous expressions from 29 participants with different ages, genders, ethnic backgrounds. Differing from existing approaches, we designed a protocol to capture the EEG signals by evoking participants' individual action units explicitly. We also investigated the relation between the EEG signals and facial action units. As a baseline, the database has been evaluated through the experiments on both posed and spontaneous emotion recognition with images alone, EEG alone, and EEG fused with images, respectively. The database will be released to the research community to advance the state of the art for automatic emotion recognition.