mlsanityData to 7 Oct 2026

Make Acoustic and Visual Cues Matter: CH-SIMS v2.0 Dataset and AV-Mixup Consistent Module

arXiv 2209.02604 ↗First posted 22 Aug 2022cs.MM cs.AI cs.CVcited by 43

Not on the map. Why?

Abstract (arXiv)

Multimodal sentiment analysis (MSA), which supposes to improve text-based sentiment analysis with associated acoustic and visual modalities, is an emerging research area due to its potential applications in Human-Computer Interaction (HCI). However, the existing researches observe that the acoustic and visual modalities contribute much less than the textual modality, termed as text-predominant. Under such circumstances, in this work, we emphasize making non-verbal cues matter for the MSA task. Firstly, from the resource perspective, we present the CH-SIMS v2.0 dataset, an extension and enhancement of the CH-SIMS. Compared with the original dataset, the CH-SIMS v2.0 doubles its size with another 2121 refined video segments with both unimodal and multimodal annotations and collects 10161 unlabelled raw video segments with rich acoustic and visual emotion-bearing context to highlight non-verbal cues for sentiment prediction. Secondly, from the model perspective, benefiting from the unimodal annotations and the unsupervised data in the CH-SIMS v2.0, the Acoustic Visual Mixup Consistent (AV-MC) framework is proposed. The designed modality mixup module can be regarded as an augmentation, which mixes the acoustic and visual modalities from different videos. Through drawing unobserved multimodal context along with the text, the model can learn to be aware of different non-verbal contexts for sentiment prediction. Our evaluations demonstrate that both CH-SIMS v2.0 and AV-MC framework enables further research for discovering emotion-bearing acoustic and visual cues and paves the path to interpretable end-to-end HCI applications for real-world scenarios.

Key facts. Cited by 43 papers in the map (data through 7 Oct 2026). Builds on 30 of its 49 references that are in the map. First posted on arXiv 22 Aug 2022.

Builds on

30 of 49 references.

References · 30

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2017mixup: Beyond Empirical Risk Minimizationcited by 5,155
  3. 2019CutMix: Regularization Strategy to Train Strong Classifiers with Localizable Featurescited by 2,875
  4. 2017Multimodal Machine Learning: A Survey and Taxonomycited by 1,004
  5. 2016Joint Face Detection and Alignment using Multi-task Cascaded Convolutional Networkscited by 895
  6. 2019Multimodal Transformer for Unaligned Multimodal Language Sequencescited by 637
  7. 2018MELD: A Multimodal Multi-Party Dataset for Emotion Recognition in Conversationscited by 636
  8. 2017Tensor Fusion Network for Multimodal Sentiment Analysiscited by 429
  9. 2018Efficient Low-rank Multimodal Fusion with Modality-Specific Factorscited by 268
  10. 2020MISA: Modality-Invariant and -Specific Representations for Multimodal Sentiment Analysiscited by 262
  11. 2022Balanced Multimodal Learning via On-the-fly Gradient Modulationcited by 259
  12. 2019Interpolation Consistency Training for Semi-Supervised Learningcited by 248
18 more
  1. 2020MixText: Linguistically-Informed Interpolation of Hidden Space for Semi-Supervised Text Classificationcited by 186
  2. 2021Learning Modality-Specific Representations with Self-Supervised Multi-Task Learning for Multimodal Sentiment Analysiscited by 179
  3. 2018Memory Fusion Network for Multi-view Sequential Learningcited by 167
  4. 2020SaliencyMix: A Saliency Guided Data Augmentation Strategy for Better Regularizationcited by 149
  5. 2021MultiBench: Multiscale Benchmarks for Multimodal Representation Learningcited by 143
  6. 2021Improving Multimodal Fusion with Hierarchical Mutual Information Maximization for Multimodal Sentiment Analysiscited by 130
  7. 2019Augmenting Data with Mixup for Sentence Classification: An Empirical Studycited by 125
  8. 2021Is Someone Speaking? Exploring Long-term Temporal Features for Audio-visual Active Speaker Detectioncited by 110
  9. 2018Words Can Shift: Dynamically Adjusting Word Representations Using Nonverbal Behaviorscited by 89
  10. 2020Beneath the Tip of the Iceberg: Current Challenges and New Directions in Sentiment Analysis Researchcited by 67
  11. 2021SSMix: Saliency-Based Span Mixup for Text Classificationcited by 39
  12. 2018Multimodal Sentiment Analysis: Addressing Key Issues and Setting up the Baselinescited by 23
  13. 2022M-SENA: An Integrated Platform for Multimodal Sentiment Analysiscited by 21
  14. 2021ScaleVLAD: Improving Multimodal Sentiment Analysis via Multi-Scale Fusion of Locally Descriptorscited by 12
  15. 2020Utterance-level Dialogue Understanding: An Empirical Studycited by 10
  16. 2021Data Augmentation in a Hybrid Approach for Aspect-Based Sentiment Analysiscited by 10
  17. 2022Sentiment Word Aware Multimodal Refinement for Multimodal Sentiment Analysis with ASR Errorscited by 7
  18. 2021On the Impact of Word Error Rate on Acoustic-Linguistic Speech Emotion Recognition: An Update for the Deep Learning Eracited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 1; 2024: 9; 2025: 14; 2026: 19.

43 papers.

Sort:

By area:

  1. 2024Explainable and Interpretable Multimodal Large Language Models: A Comprehensive Surveycited by 52
  2. 2025AffectGPT: A New Dataset, Model, and Benchmark for Emotion Understanding with Multimodal Large Language Modelscited by 43
  3. 2025EmoBench-M: Benchmarking Emotional Intelligence for Multimodal Large Language Modelscited by 32
  4. 2024Coupled Mamba: Enhanced Multi-modal Fusion with Coupled State Space Modelcited by 30
  5. 2025MME-Emotion: A Holistic Evaluation Benchmark for Emotional Intelligence in Multimodal Large Language Modelscited by 17
  6. 2024Large Language Models Meet Text-Centric Multimodal Sentiment Analysis: A Surveycited by 16
  7. 2024Recent Advances in Multimodal Affective Computing: An NLP Perspectivecited by 12
  8. 2025MSE-Adapter: A Lightweight Plugin Endowing LLMs with the Capability to Perform Multimodal Sentiment Analysis and Emotion Recognitioncited by 9
  9. 2025Can Large Language Models Help Multimodal Language Analysis? MMLA: A Comprehensive Benchmarkcited by 8
  10. 2024Knowledge-Guided Dynamic Modality Attention Fusion Framework for Multimodal Sentiment Analysiscited by 7
  11. 2025Improving Multimodal Sentiment Analysis via Modality Optimization and Dynamic Primary Modality Selectioncited by 6
  12. 2026OmniOPSD: Rationale-Privileged On-Policy Self-Distillation for Affective Computingcited by 6
31 more
  1. 2026Emotion-LLaMAv2 and MMEVerse: A New Framework and Benchmark for Multimodal Emotion Understandingcited by 5
  2. 2024End-to-end Semantic-centric Video-based Multimodal Affective Computingcited by 4
  3. 2025Fusion or Confusion? Multimodal Complexity Is Not All You Needcited by 4
  4. 2025DeepMLF: Multimodal language model with learnable tokens for deep fusion in sentiment analysiscited by 3
  5. 2025Towards Explainable Fusion and Balanced Learning in Multimodal Sentiment Analysiscited by 3
  6. 2025Multimodal Large Language Models for End-to-End Affective Computing: Benchmarking and Boosting with Generative Knowledge Promptingcited by 3
  7. 2026Navigating the Emotion Tree: Hierarchical Hyperbolic RAG for Multimodal Emotion Recognitioncited by 3
  8. 2024Evaluation of data inconsistency for multi-modal sentiment analysiscited by 2
  9. 2026Complementarity-Supervised Spectral-Band Routing for Multimodal Emotion Recognitioncited by 2
  10. 2023PowMix: A Versatile Regularizer for Multimodal Sentiment Analysiscited by 1
  11. 2024Enhancing Multimodal Affective Analysis with Learned Live Comment Featurescited by 1
  12. 2025Calibrating Multimodal Consensus for Emotion Recognitioncited by 1
  13. 2025MULTIBENCH++: A Unified and Comprehensive Multimodal Fusion Benchmarking Across Specialized Domainscited by 1
  14. 2026CaReFlow: Cyclic Adaptive Rectified Flow for Multimodal Fusioncited by 1
  15. 2026Nano-EmoX: Unifying Multimodal Emotional Intelligence from Perception to Empathycited by 1
  16. 2026EmoMM: Benchmarking and Steering MLLM for Multimodal Emotion Recognition under Conflict and Missingnesscited by 1
  17. 2026EmoS: A High-Fidelity Multimodal Benchmark for Fine-grained Streaming Emotional Understandingcited by 1
  18. 2024SentiXRL: An advanced large language Model Framework for Multilingual Fine-Grained Emotion Classification in Complex Text Environmentcited by 0
  19. 2025OmniMER: Auxiliary-Enhanced LLM Adaptation for Indonesian Multimodal Emotion Recognitioncited by 0
  20. 2025Text-Routed Sparse Mixture-of-Experts Model with Explanation and Temporal Alignment for Multi-Modal Sentiment Analysiscited by 0
  21. 2026Expert-Guided Multimodal Fusion for Unified Emotion and Sentiment Analysiscited by 0
  22. 2026GRCF: Two-Stage Groupwise Ranking and Calibration Framework for Multimodal Sentiment Analysiscited by 0
  23. 2026Integrating Fine-Grained Audio-Visual Evidence for Robust Multimodal Emotion Reasoningcited by 0
  24. 2026Learning Invariant Modality Representation for Robust Multimodal Learning from a Causal Inference Perspectivecited by 0
  25. 2026To Fuse or to Drop? Dual-Path Learning for Resolving Modality Conflicts in Multimodal Emotion Recognitioncited by 0
  26. 2026MAF: Multimodal Adaptive Few-shot Prompting for Sentiment Analysis with MLLMscited by 0
  27. 2026Do We Really Need Multimodal Emotion Language Models Larger Than 1B Parameters?cited by 0
  28. 2026Multi-Granularity Sentiment Integration for LLM-Based Multimodal Sentiment Analysiscited by 0
  29. 2026MVFA: A Multi-View Text-Guided Multimodal Fusion LLM Adapter for Sentiment Analysis and Emotion Recognitioncited by 0
  30. 2026Fine-Grained Visual Preprocessing and Dual-Stream Temporal Modeling for Multimodal Sentiment Analysis on Social Mediacited by 0
  31. 2026VISTA: Value-Informed Event Appraisal for Multimodal Emotion Conflictcited by 0