# Multi-view Multi-label Fine-grained Emotion Decoding from Human Brain Activity

arXiv: https://arxiv.org/abs/2211.02629 · HTML: https://mlsanity.com/paper/2211.02629/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 13 of its 54 references that are in the map.
- First posted on arXiv 26 Oct 2022.

## Builds on (most cited first, 13 of 13)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Generating Sentences from a Continuous Space](https://mlsanity.com/paper/1511.06349/) (2015)
- [GoEmotions: A Dataset of Fine-Grained Emotions](https://mlsanity.com/paper/2005.00547/) (2020)
- [Asymmetric Loss For Multi-Label Classification](https://mlsanity.com/paper/2009.14119/) (2020)
- [Multi-Label Image Recognition with Graph Convolutional Networks](https://mlsanity.com/paper/1904.03582/) (2019)
- [Multimodal Generative Models for Scalable Weakly-Supervised Learning](https://mlsanity.com/paper/1802.05335/) (2018)
- [Query2Label: A Simple Transformer Way to Multi-Label Classification](https://mlsanity.com/paper/2107.10834/) (2021)
- [Generalized Product of Experts for Automatic and Principled Fusion of Gaussian Process Predictions](https://mlsanity.com/paper/1410.7827/) (2014)
- [From voxels to pixels and back: Self-supervision in natural-image reconstruction from fMRI](https://mlsanity.com/paper/1907.02431/) (2019)
- [Contrastive Learning of Subject-Invariant EEG Representations for Cross-Subject Emotion Recognition](https://mlsanity.com/paper/2109.09559/) (2021)
- [A Novel Bi-hemispheric Discrepancy Model for EEG Emotion Recognition](https://mlsanity.com/paper/1906.01704/) (2019)
- [Semi-supervised Deep Generative Modelling of Incomplete Multi-Modality Emotional Data](https://mlsanity.com/paper/1808.02096/) (2018)
- [Multi-Label Activity Recognition using Activity-specific Features and Activity Correlations](https://mlsanity.com/paper/2009.07420/) (2020)

## Built on by (most cited first, 3 of 3)

- [Decoding Natural Images from EEG for Object Recognition](https://mlsanity.com/paper/2308.13234/) (2023)
- [Memory-guided Prototypical Co-occurrence Learning for Mixed Emotion Recognition](https://mlsanity.com/paper/2602.20530/) (2026)
- [Multi-label Class Incremental Emotion Decoding with Augmented Emotional Semantics Learning](https://mlsanity.com/paper/2405.20600/) (2024)

## Abstract (arXiv)

Decoding emotional states from human brain activity plays an important role in brain-computer interfaces. Existing emotion decoding methods still have two main limitations: one is only decoding a single emotion category from a brain activity pattern and the decoded emotion categories are coarse-grained, which is inconsistent with the complex emotional expression of human; the other is ignoring the discrepancy of emotion expression between the left and right hemispheres of human brain. In this paper, we propose a novel multi-view multi-label hybrid model for fine-grained emotion decoding (up to 80 emotion categories) which can learn the expressive neural representations and predicting multiple emotional states simultaneously. Specifically, the generative component of our hybrid model is parametrized by a multi-view variational auto-encoder, in which we regard the brain activity of left and right hemispheres and their difference as three distinct views, and use the product of expert mechanism in its inference network. The discriminative component of our hybrid model is implemented by a multi-label classification network with an asymmetric focal loss. For more accurate emotion decoding, we first adopt a label-aware module for emotion-specific neural representations learning and then model the dependency of emotional states by a masked self-attention mechanism. Extensive experiments on two visually evoked emotional datasets show the superiority of our method.