mlsanityData to 7 Oct 2026

Recent Advances in Multimodal Affective Computing: An NLP Perspective

arXiv 2409.07388 ↗First posted 11 Sep 2024cs.CLcited by 12

Not on the map. Why?

Abstract (arXiv)

Multimodal affective computing has gained increasing attention due to its broad applications in understanding human behavior and intentions, particularly in text-centric multimodal scenarios. Existing research spans diverse tasks, modalities, and modeling paradigms, yet lacks a unified perspective. In this survey, we systematically review recent advances from an NLP perspective, focusing on four representative tasks: multimodal sentiment analysis (MSA), multimodal emotion recognition in conversation (MERC), multimodal aspect-based sentiment analysis (MABSA), and multimodal multi-label emotion recognition (MMER). We present a unified view by comparing task formulations, benchmark datasets, and evaluation protocols, and by organizing representative methods into key paradigms, including multitask learning, pre-trained models, knowledge enhancement, and contextual modeling. We further extend the discussion to related directions, such as facial, acoustic, and physiological modalities, as well as emotion cause analysis. Finally, we highlight key challenges and outline promising future directions. To facilitate further research, we release a curated repository of relevant works and resources footnotehttps://anonymous.4open.science/r/Multimodal-Affective-Computing-Survey-9819.

Key facts. Cited by 12 papers in the map (data through 7 Oct 2026). Builds on 124 of its 310 references that are in the map. First posted on arXiv 11 Sep 2024.

Builds on

124 of 310 references.

References · 124

  1. 2017Attention Is All You Needcited by 59,932
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  5. 2023GPT-4 Technical Reportcited by 18,132
  6. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  7. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  8. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  9. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  10. 2023Visual Instruction Tuningcited by 7,792
  11. 2019EfficientNet: Rethinking Model Scaling for Convolutional Neural Networkscited by 6,415
  12. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
38 more
  1. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,874
  2. 2023Mamba: Linear-Time Sequence Modeling with Selective State Spacescited by 4,616
  3. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  4. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,195
  5. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 3,972
  6. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,878
  7. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,853
  8. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
  9. 2013Distributed Representations of Words and Phrases and their Compositionalitycited by 3,462
  10. 2023Mistral 7Bcited by 3,065
  11. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,855
  12. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,301
  13. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,219
  14. 2023Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyondcited by 2,179
  15. 2021Align before Fuse: Vision and Language Representation Learning with Momentum Distillationcited by 1,612
  16. 2022CoCa: Contrastive Captioners are Image-Text Foundation Modelscited by 1,265
  17. 2016ConceptNet 5.5: An Open Multilingual Graph of General Knowledgecited by 1,065
  18. 2017Multimodal Machine Learning: A Survey and Taxonomycited by 1,004
  19. 2021Towards a Unified View of Parameter-Efficient Transfer Learningcited by 863
  20. 2017A Survey on Multi-Task Learningcited by 838
  21. 2019Multimodal Transformer for Unaligned Multimodal Language Sequencescited by 637
  22. 2018MELD: A Multimodal Multi-Party Dataset for Emotion Recognition in Conversationscited by 636
  23. 2021AST: Audio Spectrogram Transformercited by 634
  24. 2022Image as a Foreign Language: BEiT Pretraining for All Vision and Vision-Language Taskscited by 510
  25. 2019COMET: Commonsense Transformers for Automatic Knowledge Graph Constructioncited by 450
  26. 2017Tensor Fusion Network for Multimodal Sentiment Analysiscited by 429
  27. 2021VATT: Transformers for Multimodal Self-Supervised Learning from Raw Video, Audio and Textcited by 427
  28. 2021VLMo: Unified Vision-Language Pre-Training with Mixture-of-Modality-Expertscited by 424
  29. 2020Meta Pseudo Labelscited by 372
  30. 2020MISA: Modality-Invariant and -Specific Representations for Multimodal Sentiment Analysiscited by 262
  31. 2020UniVL: A Unified Video and Language Pre-Training Model for Multimodal Understanding and Generationcited by 258
  32. 2023Multimodal Deep Learningcited by 257
  33. 2022UnifiedSKG: Unifying and Multi-Tasking Structured Knowledge Grounding with Text-to-Text Language Modelscited by 231
  34. 2022Challenges and Strategies in Cross-Cultural NLPcited by 216
  35. 2020Mutual Mean-Teaching: Pseudo Label Refinery for Unsupervised Domain Adaptation on Person Re-identificationcited by 214
  36. 2019DialogueGCN: A Graph Convolutional Neural Network for Emotion Recognition in Conversationcited by 184
  37. 2021Learning Modality-Specific Representations with Self-Supervised Multi-Task Learning for Multimodal Sentiment Analysiscited by 179
  38. 2021Improving Multimodal Fusion with Hierarchical Mutual Information Maximization for Multimodal Sentiment Analysiscited by 130

50 most cited of 124 All 124 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 7; 2026: 4.

12 papers.

  1. 2025VistaDPO: Video Hierarchical Spatial-Temporal Direct Preference Optimization for Large Video Modelscited by 14
  2. 2025EvoEmpirBench: Dynamic Spatial Reasoning with Agent-ExpVercited by 5
  3. 2025GatedxLSTM: A Multimodal Affective Computing Approach for Emotion Recognition in Conversationscited by 3
  4. 2024ExpressivityBench: Can LLMs Communicate Implicitly?cited by 2
  5. 2025When LLMs Team Up: The Emergence of Collaborative Affective Computingcited by 2
  6. 2025Long-Short Distance Graph Neural Networks and Improved Curriculum Learning for Emotion Recognition in Conversationcited by 2
  7. 2025Do LLMs Feel? Teaching Emotion Recognition with Prompts, Retrieval, and Curriculum Learningcited by 2
  8. 2026STAR: Mitigating Cascading Errors in Spatial Reasoning via Turn-point Alignment and Segment-level DPOcited by 2
  9. 2026MissBench: Benchmarking Multimodal Affective Analysis under Imbalanced Missing Modalitiescited by 1
  10. 2025Structure-Aware Prototype Guided Trusted Multi-View Classificationcited by 0
  11. 2026Backdoor Learning in Language Models and Vision-Language Modelscited by 0
  12. 2026AffectOmni: RL-Verifiable People-Centric Grounded Affective Reasoning for Social and Art-Related Scenescited by 0