On the Use of Modality-Specific Large-Scale Pre-Trained Encoders for Multimodal Sentiment Analysis
Not on the map. Why?
Abstract (arXiv)
This paper investigates the effectiveness and implementation of modality-specific large-scale pre-trained encoders for multimodal sentiment analysis~(MSA). Although the effectiveness of pre-trained encoders in various fields has been reported, conventional MSA methods employ them for only linguistic modality, and their application has not been investigated. This paper compares the features yielded by large-scale pre-trained encoders with conventional heuristic features. One each of the largest pre-trained encoders publicly available for each modality are used; CLIP-ViT, WavLM, and BERT for visual, acoustic, and linguistic modalities, respectively. Experiments on two datasets reveal that methods with domain-specific pre-trained encoders attain better performance than those with conventional features in both unimodal and multimodal scenarios. We also find it better to use the outputs of the intermediate layers of the encoders than those of the output layer. The codes are available at https://github.com/ando-hub/MSA_Pretrain.
Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Builds on 20 of its 29 references that are in the map. First posted on arXiv 28 Oct 2022.
Builds on
20 of 29 references.
References · 20
- 2017Attention Is All You Needcited by 60,361
- 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
- 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
- 2020Language Models are Few-Shot Learnerscited by 32,419
- 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 12,049
- 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,905
- 2019XLNet: Generalized Autoregressive Pretraining for Language Understandingcited by 2,811
- 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,342
- 2021WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processingcited by 1,657
- 2019SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognitioncited by 1,380
- 2016Joint Face Detection and Alignment using Multi-task Cascaded Convolutional Networkscited by 903
- 2019Multimodal Transformer for Unaligned Multimodal Language Sequencescited by 645
8 more
- 2017Tensor Fusion Network for Multimodal Sentiment Analysiscited by 433
- 2020MISA: Modality-Invariant and -Specific Representations for Multimodal Sentiment Analysiscited by 265
- 2021Learning Modality-Specific Representations with Self-Supervised Multi-Task Learning for Multimodal Sentiment Analysiscited by 180
- 2018Learning Factorized Multimodal Representationscited by 171
- 2018Found in Translation: Learning Robust Joint Representations by Cyclic Translations Between Modalitiescited by 149
- 2018Multi-attention Recurrent Network for Human Communication Comprehensioncited by 89
- 2018Words Can Shift: Dynamically Adjusting Word Representations Using Nonverbal Behaviorscited by 89
- 2018Multimodal Language Analysis with Recurrent Multistage Fusioncited by 44
Built on by
Citing papers per year: 2023: 1; 2024: 0; 2025: 2.
3 papers.