# Emotion Embeddings unicodex2014 Learning Stable and Homogeneous Abstractions from Heterogeneous Affective Datasets

arXiv: https://arxiv.org/abs/2308.07871 · HTML: https://mlsanity.com/paper/2308.07871/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 49 of its 158 references that are in the map.
- First posted on arXiv 15 Aug 2023.

## Builds on (most cited first, 30 of 49)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [HuggingFace's Transformers: State-of-the-art Natural Language Processing](https://mlsanity.com/paper/1910.03771/) (2019)
- [An Overview of Multi-Task Learning in Deep Neural Networks](https://mlsanity.com/paper/1706.05098/) (2017)
- [How multilingual is Multilingual BERT?](https://mlsanity.com/paper/1906.01502/) (2019)
- [MELD: A Multimodal Multi-Party Dataset for Emotion Recognition in Conversations](https://mlsanity.com/paper/1810.02508/) (2018)
- [AffectNet: A Database for Facial Expression, Valence, and Arousal Computing in the Wild](https://mlsanity.com/paper/1708.03985/) (2017)
- [Tensor Fusion Network for Multimodal Sentiment Analysis](https://mlsanity.com/paper/1707.07250/) (2017)
- [Challenges in Representation Learning: A report on three machine learning contests](https://mlsanity.com/paper/1307.0414/) (2013)
- [Learning Word Vectors for 157 Languages](https://mlsanity.com/paper/1802.06893/) (2018)
- [Deep Facial Expression Recognition: A Survey](https://mlsanity.com/paper/1804.08348/) (2018)
- [Emotion Recognition in Conversation: Research Challenges, Datasets, and Recent Advances](https://mlsanity.com/paper/1905.02947/) (2019)
- [Using millions of emoji occurrences to learn any-domain representations for detecting sentiment, emotion and sarcasm](https://mlsanity.com/paper/1708.00524/) (2017)
- [Knowledge-Enriched Transformer for Emotion Detection in Textual Conversations](https://mlsanity.com/paper/1909.10681/) (2019)
- [EmoBank: Studying the Impact of Annotation Perspective and Representation Format on Dimensional Emotion Analysis](https://mlsanity.com/paper/2205.01996/) (2022)
- [Context Based Emotion Recognition using EMOTIC Dataset](https://mlsanity.com/paper/2003.13401/) (2020)
- [End-to-End Multimodal Emotion Recognition using Deep Neural Networks](https://mlsanity.com/paper/1704.08619/) (2017)
- [A Systematic Review on Affective Computing: Emotion Models, Databases, and Recent Advances](https://mlsanity.com/paper/2203.06935/) (2022)
- [Facial Expression Recognition with Visual Transformers and Attentional Selective Fusion](https://mlsanity.com/paper/2103.16854/) (2021)
- [Survey on Emotional Body Gesture Recognition](https://mlsanity.com/paper/1801.07481/) (2018)
- [Affective Image Content Analysis: Two Decades Review and New Perspectives](https://mlsanity.com/paper/2106.16125/) (2021)
- [A Transformer-based joint-encoding for Emotion Recognition and Sentiment Analysis](https://mlsanity.com/paper/2006.15955/) (2020)
- [HiGRU: Hierarchical Gated Recurrent Units for Utterance-level Emotion Recognition](https://mlsanity.com/paper/1904.04446/) (2019)
- [Multi-task Learning for Multi-modal Emotion Recognition and Sentiment Analysis](https://mlsanity.com/paper/1905.05812/) (2019)
- [Jointly Fine-Tuning "BERT-like" Self Supervised Models to Improve Multimodal Speech Emotion Recognition](https://mlsanity.com/paper/2008.06682/) (2020)
- [Vision Transformer with Attentive Pooling for Robust Facial Expression Recognition](https://mlsanity.com/paper/2212.05463/) (2022)
- [NRC-Canada: Building the State-of-the-Art in Sentiment Analysis of Tweets](https://mlsanity.com/paper/1308.6242/) (2013)
- [Sentiment of Emojis](https://mlsanity.com/paper/1509.07761/) (2015)
- [MDAN: Multi-level Dependent Attention Network for Visual Emotion Analysis](https://mlsanity.com/paper/2203.13443/) (2022)
- [Appraisal Theories for Emotion Classification in Text](https://mlsanity.com/paper/2003.14155/) (2020)

## Abstract (arXiv)

Human emotion is expressed in many communication modalities and media formats and so their computational study is equally diversified into natural language processing, audio signal analysis, computer vision, etc. Similarly, the large variety of representation formats used in previous research to describe emotions (polarity scales, basic emotion categories, dimensional approaches, appraisal theory, etc.) have led to an ever proliferating diversity of datasets, predictive models, and software tools for emotion analysis. Because of these two distinct types of heterogeneity, at the expressional and representational level, there is a dire need to unify previous work on increasingly diverging data and label types. This article presents such a unifying computational model. We propose a training procedure that learns a shared latent representation for emotions, so-called emotion embeddings, independent of different natural languages, communication modalities, media or representation label formats, and even disparate model architectures. Experiments on a wide range of heterogeneous affective datasets indicate that this approach yields the desired interoperability for the sake of reusability, interpretability and flexibility, without penalizing prediction quality. Code and data are archived under https://doi.org/10.5281/zenodo.7405327 .