# The Multimodal Sentiment Analysis in Car Reviews (MuSe-CaR) Dataset: Collection, Insights and Improvements

arXiv: https://arxiv.org/abs/2101.06053 · HTML: https://mlsanity.com/paper/2101.06053/

## Key facts

- Cited by 22 papers in the map (data through 14 Oct 2026).
- Builds on 33 of its 110 references that are in the map.
- First posted on arXiv 15 Jan 2021.

## Builds on (most cited first, 30 of 33)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding](https://mlsanity.com/paper/1804.07461/) (2018)
- [Deep Learning Face Attributes in the Wild](https://mlsanity.com/paper/1411.7766/) (2014)
- [Distributed Representations of Words and Phrases and their Compositionality](https://mlsanity.com/paper/1310.4546/) (2013)
- [Enriching Word Vectors with Subword Information](https://mlsanity.com/paper/1607.04606/) (2016)
- [Multimodal Machine Learning: A Survey and Taxonomy](https://mlsanity.com/paper/1705.09406/) (2017)
- [Joint Face Detection and Alignment using Multi-task Cascaded Convolutional Networks](https://mlsanity.com/paper/1604.02878/) (2016)
- [CNN Architectures for Large-Scale Audio Classification](https://mlsanity.com/paper/1609.09430/) (2016)
- [Tensor Fusion Network for Multimodal Sentiment Analysis](https://mlsanity.com/paper/1707.07250/) (2017)
- [Deep Audio-Visual Speech Recognition](https://mlsanity.com/paper/1809.02108/) (2018)
- [Deep Facial Expression Recognition: A Survey](https://mlsanity.com/paper/1804.08348/) (2018)
- [Expression, Affect, Action Unit Recognition: Aff-Wild2, Multi-Task Learning and ArcFace](https://mlsanity.com/paper/1910.04855/) (2019)
- [Deep Affect Prediction in-the-wild: Aff-Wild Database and Challenge, Deep Architectures, and Beyond](https://mlsanity.com/paper/1804.10938/) (2018)
- [MOSI: Multimodal Corpus of Sentiment Intensity and Subjectivity Analysis in Online Opinion Videos](https://mlsanity.com/paper/1606.06259/) (2016)
- [Analysing Affective Behavior in the First ABAW 2020 Competition](https://mlsanity.com/paper/2001.11409/) (2020)
- [UR-FUNNY: A Multimodal Language Dataset for Understanding Humor](https://mlsanity.com/paper/1904.06618/) (2019)
- [Exploring Hate Speech Detection in Multimodal Publications](https://mlsanity.com/paper/1910.03814/) (2019)
- [Multi-attention Recurrent Network for Human Communication Comprehension](https://mlsanity.com/paper/1802.00923/) (2018)
- [SEWA DB: A Rich Database for Audio-Visual Emotion and Sentiment Research in the Wild](https://mlsanity.com/paper/1901.02839/) (2019)
- [Sentiment Analysis: Automatically Detecting Valence, Emotions, and Other Affectual States from Text](https://mlsanity.com/paper/2005.11882/) (2020)
- [The Evolution of Sentiment Analysis - A Review of Research Topics, Venues, and Top Cited Papers](https://mlsanity.com/paper/1612.01556/) (2016)
- [On the Role of Text Preprocessing in Neural Network Architectures: An Evaluation Study on Text Categorization and Sentiment Analysis](https://mlsanity.com/paper/1707.01780/) (2017)
- [Learning Intelligent Dialogs for Bounding Box Annotation](https://mlsanity.com/paper/1712.08087/) (2017)
- [Text Flow: A Unified Text Detection System in Natural Scene Images](https://mlsanity.com/paper/1604.06877/) (2016)
- [The Many-to-Many Mapping Between the Concordance Correlation Coefficient and the Mean Square Error](https://mlsanity.com/paper/1902.05180/) (2019)
- [Towards Multimodal Emotion Recognition in German Speech Events in Cars using Transfer Learning](https://mlsanity.com/paper/1909.02764/) (2019)
- [Mining fine-grained opinions on closed captions of YouTube videos with an attention-RNN](https://mlsanity.com/paper/1708.02420/) (2017)
- [Visual and Textual Sentiment Analysis Using Deep Fusion Convolutional Neural Networks](https://mlsanity.com/paper/1711.07798/) (2017)
- [A multimodal movie review corpus for fine-grained opinion mining](https://mlsanity.com/paper/1902.10102/) (2019)
- [Extracting textual overlays from social media videos using neural networks](https://mlsanity.com/paper/1804.10687/) (2018)

## Built on by (most cited first, 22 of 22)

- [Hybrid Contrastive Learning of Tri-Modal Representation for Multimodal Sentiment Analysis](https://mlsanity.com/paper/2109.01797/) (2021)
- [The MuSe 2021 Multimodal Sentiment Analysis Challenge: Sentiment, Emotion, Physiological-Emotion, and Stress](https://mlsanity.com/paper/2104.07123/) (2021)
- [Multimodality Representation Learning: A Survey on Evolution, Pretraining and Its Applications](https://mlsanity.com/paper/2302.00389/) (2023)
- [The MuSe 2022 Multimodal Sentiment Analysis Challenge: Humor, Emotional Reactions, and Stress](https://mlsanity.com/paper/2207.05691/) (2022)
- [Efficient Multimodal Transformer with Dual-Level Feature Restoration for Robust Multimodal Sentiment Analysis](https://mlsanity.com/paper/2208.07589/) (2022)
- [Continuous Emotion Recognition with Audio-visual Leader-follower Attentive Fusion](https://mlsanity.com/paper/2107.01175/) (2021)
- [Recent Advances in Multimodal Affective Computing: An NLP Perspective](https://mlsanity.com/paper/2409.07388/) (2024)
- [Multimodal Sentiment Analysis: A Survey](https://mlsanity.com/paper/2305.07611/) (2023)
- [CI-AVSR: A Cantonese Audio-Visual Speech Dataset for In-car Command Recognition](https://mlsanity.com/paper/2201.03804/) (2022)
- [MuSe-Toolbox: The Multimodal Sentiment Analysis Continuous Annotation Fusion and Discrete Class Transformation Toolbox](https://mlsanity.com/paper/2107.11757/) (2021)
- [PTSD in the Wild: A Video Database for Studying Post-Traumatic Stress Disorder Recognition in Unconstrained Environments](https://mlsanity.com/paper/2209.14085/) (2022)
- [Affective Video Content Analysis: Decade Review and New Perspectives](https://mlsanity.com/paper/2310.17212/) (2023)
- [Hybrid Mutimodal Fusion for Dimensional Emotion Recognition](https://mlsanity.com/paper/2110.08495/) (2021)
- [Computational Analysis of Stress, Depression and Engagement in Mental Health: A Survey](https://mlsanity.com/paper/2403.08824/) (2024)
- [A Comprehensive Review on Sentiment Analysis: Tasks, Approaches and Applications](https://mlsanity.com/paper/2311.11250/) (2023)
- [RoHyDR: Robust Hybrid Diffusion Recovery for Incomplete Multimodal Emotion Recognition](https://mlsanity.com/paper/2505.17501/) (2025)
- [GraphTMT: Unsupervised Graph-based Topic Modeling from Video Transcripts](https://mlsanity.com/paper/2105.01466/) (2021)
- [An Estimation of Online Video User Engagement from Features of Continuous Emotions](https://mlsanity.com/paper/2105.01633/) (2021)
- [Sequential Late Fusion Technique for Multi-modal Sentiment Analysis](https://mlsanity.com/paper/2106.11473/) (2021)
- [PowMix: A Versatile Regularizer for Multimodal Sentiment Analysis](https://mlsanity.com/paper/2312.12334/) (2023)
- [UL-DD: A Multimodal Drowsiness Dataset Using Video, Biometric Signals, and Behavioral Data](https://mlsanity.com/paper/2507.13403/) (2025)
- [A Data-to-Product Multimodal Conceptual Framework to Achieve Automated Software Evolution for Context-rich Intelligent Applications](https://mlsanity.com/paper/2404.04821/) (2024)

## Abstract (arXiv)

Truly real-life data presents a strong, but exciting challenge for sentiment and emotion research. The high variety of possible `in-the-wild' properties makes large datasets such as these indispensable with respect to building robust machine learning models. A sufficient quantity of data covering a deep variety in the challenges of each modality to force the exploratory analysis of the interplay of all modalities has not yet been made available in this context. In this contribution, we present MuSe-CaR, a first of its kind multimodal dataset. The data is publicly available as it recently served as the testing bed for the 1st Multimodal Sentiment Analysis Challenge, and focused on the tasks of emotion, emotion-target engagement, and trustworthiness recognition by means of comprehensively integrating the audio-visual and language modalities. Furthermore, we give a thorough overview of the dataset in terms of collection and annotation, including annotation tiers not used in this year's MuSe 2020. In addition, for one of the sub-challenges - predicting the level of trustworthiness - no participant outperformed the baseline model, and so we propose a simple, but highly efficient Multi-Head-Attention network that exceeds using multimodal fusion the baseline by around 0.2 CCC (almost 50 % improvement).