mlsanityData to 7 Oct 2026

TEASEL: A Transformer-Based Speech-Prefixed Language Model

arXiv 2109.05522 ↗First posted 12 Sep 2021cs.CL cs.AI cs.LGcited by 8

Not on the map. Why?

Abstract (arXiv)

Multimodal language analysis is a burgeoning field of NLP that aims to simultaneously model a speaker's words, acoustical annotations, and facial expressions. In this area, lexicon features usually outperform other modalities because they are pre-trained on large corpora via Transformer-based models. Despite their strong performance, training a new self-supervised learning (SSL) Transformer on any modality is not usually attainable due to insufficient data, which is the case in multimodal language learning. This work proposes a Transformer-Based Speech-Prefixed Language Model called TEASEL to approach the mentioned constraints without training a complete Transformer model. TEASEL model includes speech modality as a dynamic prefix besides the textual modality compared to a conventional language model. This method exploits a conventional pre-trained language model as a cross-modal Transformer model. We evaluated TEASEL for the multimodal sentiment analysis task defined by CMU-MOSI dataset. Extensive experiments show that our model outperforms unimodal baseline language models by 4% and outperforms the current multimodal state-of-the-art (SoTA) model by 1% in F1-score. Additionally, our proposed method is 72% smaller than the SoTA model.

Key facts. Cited by 8 papers in the map (data through 7 Oct 2026). Builds on 34 of its 46 references that are in the map. First posted on arXiv 12 Sep 2021.

Builds on

34 of 46 references.

References · 34

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  5. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  6. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  7. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  8. 2014Neural Machine Translation by Jointly Learning to Align and Translatecited by 4,530
  9. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,853
  10. 2016Gaussian Error Linear Units (GELUs)cited by 3,431
  11. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,219
  12. 2017Multimodal Machine Learning: A Survey and Taxonomycited by 1,004
22 more
  1. 2018VoxCeleb2: Deep Speaker Recognitioncited by 862
  2. 2021Multimodal Few-Shot Learning with Frozen Language Modelscited by 657
  3. 2019Multimodal Transformer for Unaligned Multimodal Language Sequencescited by 637
  4. 2017Tensor Fusion Network for Multimodal Sentiment Analysiscited by 429
  5. 2018Efficient Low-rank Multimodal Fusion with Modality-Specific Factorscited by 268
  6. 2020MISA: Modality-Invariant and -Specific Representations for Multimodal Sentiment Analysiscited by 262
  7. 2016MOSI: Multimodal Corpus of Sentiment Intensity and Subjectivity Analysis in Online Opinion Videoscited by 193
  8. 2021Learning Modality-Specific Representations with Self-Supervised Multi-Task Learning for Multimodal Sentiment Analysiscited by 179
  9. 2019Towards Multimodal Sarcasm Detection (An _Obviously_ Perfect Paper)cited by 173
  10. 2018Learning Factorized Multimodal Representationscited by 171
  11. 2018Memory Fusion Network for Multi-view Sequential Learningcited by 167
  12. 2021Pretrained Transformers as Universal Computation Enginescited by 162
  13. 2018Found in Translation: Learning Robust Joint Representations by Cyclic Translations Between Modalitiescited by 149
  14. 2019UR-FUNNY: A Multimodal Language Dataset for Understanding Humorcited by 130
  15. 2018Multi-attention Recurrent Network for Human Communication Comprehensioncited by 89
  16. 2018Words Can Shift: Dynamically Adjusting Word Representations Using Nonverbal Behaviorscited by 89
  17. 2019Learning Relationships between Text, Audio, and Video via Deep Canonical Correlation for Multimodal Language Analysiscited by 82
  18. 2018Multimodal Sentiment Analysis with Word-Level Fusion and Reinforcement Learningcited by 43
  19. 2020Jointly Fine-Tuning "BERT-like" Self Supervised Models to Improve Multimodal Speech Emotion Recognitioncited by 38
  20. 2019Factorized Multimodal Transformer for Multimodal Sequential Learningcited by 16
  21. 2020Self-Supervised learning with cross-modal transformers for emotion recognitioncited by 12
  22. 2020MTAG: Modal-Temporal Attention Graph for Unaligned Human Multimodal Language Sequencescited by 4

Built on by

Cited by year

Citing papers per year: 2023: 2; 2024: 5; 2025: 1.

8 papers.

  1. 2023Transformers in Speech Processing: A Surveycited by 38
  2. 2024Beyond Silent Letters: Amplifying LLMs in Emotion Recognition with Vocal Nuancescited by 27
  3. 2023Multimodal Multi-loss Fusion Network for Sentiment Analysiscited by 26
  4. 2024Paralinguistics-Aware Speech-Empowered Large Language Models for Natural Conversationcited by 14
  5. 2024Recent Advances in Multimodal Affective Computing: An NLP Perspectivecited by 12
  6. 2024Survey: Transformer-based Models in Data Modality Conversioncited by 0
  7. 2024MEANT: Multimodal Encoder for Antecedent Informationcited by 0
  8. 2025Chain of Questions: Guiding Multimodal Curiosity in Language Modelscited by 0