Cross-stitched Multi-modal Encoders
Not on the map. Why?
Abstract (arXiv)
In this paper, we propose a novel architecture for multi-modal speech and text input. We combine pretrained speech and text encoders using multi-headed cross-modal attention and jointly fine-tune on the target problem. The resultant architecture can be used for continuous token-level classification or utterance-level prediction acting on simultaneous text and speech. The resultant encoder efficiently captures both acoustic-prosodic and lexical information. We compare the benefits of multi-headed attention-based fusion for multi-modal utterance-level classification against a simple concatenation of pre-pooled, modality-specific representations. Our model architecture is compact, resource efficient, and can be trained on a single consumer GPU card.
Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 33 of its 59 references that are in the map. First posted on arXiv 20 Apr 2022.
Builds on
33 of 59 references.
References · 33
- 2017Attention Is All You Needcited by 59,932
- 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
- 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
- 2014Neural Machine Translation by Jointly Learning to Align and Translatecited by 4,530
- 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,853
- 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,301
- 2018Deep contextualized word representationscited by 2,285
- 2018Self-Attention with Relative Position Representationscited by 1,112
- 2020On Layer Normalization in the Transformer Architecturecited by 906
- 2017DailyDialog: A Manually Labelled Multi-turn Dialogue Datasetcited by 649
- 2019Multimodal Transformer for Unaligned Multimodal Language Sequencescited by 637
- 2015Listen, Attend and Spellcited by 542
21 more
- 2020GoEmotions: A Dataset of Fine-Grained Emotionscited by 479
- 2015Attention-Based Models for Speech Recognitioncited by 400
- 2019vq-wav2vec: Self-Supervised Learning of Discrete Speech Representationscited by 335
- 2019Learning Deep Transformer Models for Machine Translationcited by 309
- 2019Speech Model Pre-training for End-to-End Spoken Language Understandingcited by 191
- 2020TERA: Self-Supervised Learning of Transformer Encoder Representation for Speechcited by 186
- 2019An Unsupervised Autoregressive Model for Speech Representation Learningcited by 185
- 2019Transformers without Tears: Improving the Normalization of Self-Attentioncited by 153
- 2019Learning Problem-agnostic Speech Representations from Multiple Self-supervised Taskscited by 87
- 2017Semi-supervised sequence tagging with bidirectional language modelscited by 86
- 2017End-to-End Multimodal Emotion Recognition using Deep Neural Networkscited by 69
- 2021ESPnet-SLU: Advancing Spoken Language Understanding through ESPnetcited by 53
- 2019A Repository of Conversational Datasetscited by 45
- 2021Self-supervised Contrastive Cross-Modality Representation Learning for Spoken Question Answeringcited by 39
- 2020Jointly Fine-Tuning "BERT-like" Self Supervised Models to Improve Multimodal Speech Emotion Recognitioncited by 38
- 2020End-to-end Named Entity Recognition from English Speechcited by 36
- 2016Conversational Contextual Cues: The Case of Personalization and History for Response Rankingcited by 21
- 2020ST-BERT: Cross-modal Language Model Pre-training For End-to-end Spoken Language Understandingcited by 19
- 2019LSTM based Similarity Measurement with Spectral Clustering for Speaker Diarizationcited by 17
- 2020End-to-end speech-to-dialog-act recognitioncited by 4
- 2017Parsing Speech: A Neural Approach to Integrating Lexical and Acoustic-Prosodic Informationcited by 2
Built on by
Not cited yet.