mlsanityData to 7 Oct 2026

Fused Acoustic and Text Encoding for Multimodal Bilingual Pretraining and Speech Translation

arXiv 2102.05766 ↗First posted 10 Feb 2021cs.CLcited by 47

Not on the map. Why?

Abstract (arXiv)

Recently, representation learning for text and speech has successfully improved many language related tasks. However, all existing methods suffer from two limitations: (a) they only learn from one input modality, while a unified representation for both speech and text is needed by tasks such as end-to-end speech translation, and as a result,(b) they can not exploit various large-scale text and speech data and their performance is limited by the scarcity of parallel speech translation data.To address these problems, we propose a Fused Acoustic and Text Masked Language Model (FAT-MLM) which jointly learns a unified representation for both acoustic and text input from various types of corpora including parallel data for speech recognition and machine translation, and even pure speech and text data. Within this cross-modal representation learning framework, we further present an end-to-end model for Fused Acoustic and Text Speech Translation (FAT-ST). Experiments on three translation directions show that by fine-tuning from FAT-MLM, our proposed speech translation models substantially improve translation quality by up to +5.9 BLEU.

Key facts. Cited by 47 papers in the map (data through 7 Oct 2026). Builds on 24 of its 36 references that are in the map. First posted on arXiv 10 Feb 2021.

Builds on

24 of 36 references.

References · 24

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  3. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,115
  4. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,853
  5. 2018Deep contextualized word representationscited by 2,285
  6. 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,973
  7. 2019Cross-lingual Language Model Pretrainingcited by 1,088
  8. 2019Libri-Light: A Benchmark for ASR with Limited or No Supervisioncited by 462
  9. 2019ERNIE: Enhanced Representation through Knowledge Integrationcited by 301
  10. 2020TERA: Self-Supervised Learning of Transformer Encoder Representation for Speechcited by 186
  11. 2017Sequence-to-Sequence Models Can Directly Translate Foreign Speechcited by 121
  12. 2020ESPnet-ST: All-in-One Speech Translation Toolkitcited by 82
12 more
  1. 2018End-to-End Automatic Speech Translation of Audiobookscited by 65
  2. 2019End-to-End Speech Translation with Knowledge Distillationcited by 58
  3. 2018Leveraging Weakly Supervised Data to Improve End-to-End Speech-to-Text Translationcited by 57
  4. 2020Bridging the Modality Gap for Speech-to-Text Translationcited by 55
  5. 2020Dual-decoder Transformer for Joint Automatic Speech Recognition and Multilingual Speech Translationcited by 54
  6. 2018Universal Language Model Fine-tuning for Text Classificationcited by 50
  7. 2020Self-Training for End-to-End Speech Translationcited by 36
  8. 2018Augmenting Librispeech with French Translations: A Multimodal Corpus for Direct Speech Translation Evaluationcited by 34
  9. 2020Consecutive Decoding for Speech-to-text Translationcited by 28
  10. 2017Neural Lattice-to-Sequence Models for Uncertain Inputscited by 20
  11. 2020MAM: Masked Acoustic Modeling for End-to-End Speech-to-Text Translationcited by 13
  12. 2020Fluent and Low-latency Simultaneous Speech-to-Speech Translation with Self-adaptive Trainingcited by 10

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 5; 2022: 17; 2023: 18; 2024: 2; 2025: 4; 2026: 1.

47 papers.

Sort:

By area:

  1. 2022Multimodal Learning with Transformers: A Surveycited by 399
  2. 2021SpeechT5: Unified-Modal Encoder-Decoder Pre-Training for Spoken Language Processingcited by 177
  3. 2022mSLAM: Massively multilingual joint pre-training for speech and textcited by 108
  4. 2022MAESTRO: Matched Speech Text Representations through Modality Matchingcited by 79
  5. 2021SLAM: A Unified Encoder for Speech and Language Modeling via Speech-Text Joint Pre-Trainingcited by 69
  6. 2022Building Machine Translation Systems for the Next Thousand Languagescited by 69
  7. 2022Unified Speech-Text Pre-training for Speech Translation and Recognitioncited by 64
  8. 2023ChatBridge: Bridging Modalities with Large Language Model as a Language Catalystcited by 64
  9. 2022Cross-modal Contrastive Learning for Speech Translationcited by 61
  10. 2022STEMM: Self-learning with Speech-text Manifold Mixup for Speech Translationcited by 57
  11. 2021End-to-end Speech Translation via Cross-modal Progressive Trainingcited by 47
  12. 2022A3T: Alignment-Aware Acoustic and Text Pretraining for Speech Synthesis and Editingcited by 40
35 more
  1. 2022SpeechUT: Bridging Speech and Text with Hidden-Unit for Encoder-Decoder Based Speech-Text Pre-trainingcited by 40
  2. 2023SLTUNET: A Simple Unified Model for Sign Language Translationcited by 35
  3. 2021Direct Simultaneous Speech-to-Text Translation Assisted by Synchronized Streaming ASRcited by 31
  4. 2022Revisiting End-to-End Speech-to-Text Translation From Scratchcited by 26
  5. 2023Pre-training for Speech Translation: CTC Meets Optimal Transportcited by 25
  6. 2021Searchable Hidden Intermediates for End-to-End Models of Decomposable Sequence Taskscited by 24
  7. 2023CMOT: Cross-modal Mixup via Optimal Transport for Speech Translationcited by 22
  8. 2022WACO: Word-Aligned Contrastive Learning for Speech Translationcited by 21
  9. 2022Leveraging unsupervised and weakly-supervised data to improve direct speech-to-speech translationcited by 20
  10. 2023DUB: Discrete Unit Back-translation for Speech Translationcited by 18
  11. 2022M-Adapter: Modality Adaptation for End-to-End Speech-to-Text Translationcited by 15
  12. 2023Understanding and Bridging the Modality Gap for Speech Translationcited by 15
  13. 2023Recent Advances in Direct Speech-to-text Translationcited by 15
  14. 2022Improving End-to-end Speech Translation by Leveraging Auxiliary Speech and Text Datacited by 9
  15. 2023ComSL: A Composite Speech-Language Model for End-to-End Speech-to-Text Translationcited by 9
  16. 2022Discrete Cross-Modal Alignment Enables Zero-Shot Speech Translationcited by 6
  17. 2023Understanding Shared Speech-Text Representationscited by 6
  18. 2023Rethinking and Improving Multi-task Learning for End-to-end Speech Translationcited by 6
  19. 2022An Analysis of Semantically-Aligned Speech-Text Embeddingscited by 5
  20. 2023Soft Alignment of Modality Space for End-to-end Speech Translationcited by 4
  21. 2024EMMeTT: Efficient Multimodal Machine Translation Trainingcited by 4
  22. 2025RCRank: Multimodal Ranking of Root Causes of Slow Queries in Cloud Database Systemscited by 4
  23. 2025Voice Cloning: Comprehensive Surveycited by 4
  24. 2025TouchFormer: A Robust Transformer-based Framework for Multimodal Material Perceptioncited by 4
  25. 2023Improving speech translation by fusing speech and textcited by 3
  26. 2023Towards a Deep Understanding of Multilingual End-to-End Speech Translationcited by 3
  27. 2025C2AV-TSE: Context and Confidence-aware Audio Visual Target Speaker Extractioncited by 2
  28. 2023Efficient CTC Regularization via Coarse Labels for End-to-End Speech Translationcited by 1
  29. 2023Cross-Language Speech Emotion Recognition Using Multimodal Dual Attention Transformerscited by 1
  30. 2023Improving End-to-End Speech Translation by Imitation-Based Knowledge Distillation with Synthetic Transcriptscited by 1
  31. 2024Deep Insights into Cognitive Decline: A Survey of Leveraging Non-Intrusive Modalities with Deep Learning Techniquescited by 1
  32. 2022ERNIE-SAT: Speech and Text Joint Pretraining for Cross-Lingual Multi-Speaker Text-to-Speechcited by 0
  33. 2023Hierarchical Cross-modal Transformer for RGB-D Salient Object Detectioncited by 0
  34. 2023Transformadores: Fundamentos teoricos y Aplicacionescited by 0
  35. 2026Are you speaking my languages? On spoken language adherence in multimodal LLMscited by 0