mlsanityData to 14 Oct 2026

T-Modules: Translation Modules for Zero-Shot Cross-Modal Machine Translation

arXiv 2205.12216 ↗First posted 24 May 2022cs.CLcited by 16

Not on the map. Why?

Abstract (arXiv)

We present a new approach to perform zero-shot cross-modal transfer between speech and text for translation tasks. Multilingual speech and text are encoded in a joint fixed-size representation space. Then, we compare different approaches to decode these multimodal and multilingual fixed-size representations, enabling zero-shot translation between languages and modalities. All our models are trained without the need of cross-modal labeled translation data. Despite a fixed-size representation, we achieve very competitive results on several text and speech translation tasks. In particular, we significantly improve the state-of-the-art for zero-shot speech translation on Must-C. Incorporating a speech decoder in our framework, we introduce the first results for zero-shot direct speech-to-speech and text-to-speech translation.

Key facts. Cited by 16 papers in the map (data through 14 Oct 2026). Builds on 39 of its 49 references that are in the map. First posted on arXiv 24 May 2022.

Builds on

39 of 49 references.

References · 39

  1. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,170
  2. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,905
  3. 2019Cross-lingual Language Model Pretrainingcited by 1,090
  4. 2022data2vec: A General Framework for Self-supervised Learning in Speech, Vision and Languagecited by 727
  5. 2020Language-agnostic BERT Sentence Embeddingcited by 627
  6. 2021XLS-R: Self-supervised Cross-lingual Speech Representation Learning at Scalecited by 601
  7. 2020Making Monolingual Sentence Embeddings Multilingual using Knowledge Distillationcited by 593
  8. 2020Beyond English-Centric Multilingual Machine Translationcited by 561
  9. 2021The FLORES-101 Evaluation Benchmark for Low-Resource and Multilingual Machine Translationcited by 560
  10. 2019CCNet: Extracting High Quality Monolingual Datasets from Web Crawl Datacited by 548
  11. 2020Unsupervised Cross-lingual Representation Learning for Speech Recognitioncited by 501
  12. 2021VoxPopuli: A Large-Scale Multilingual Speech Corpus for Representation Learning, Semi-Supervised Learning and Interpretationcited by 463
27 more
  1. 2018Massively Multilingual Sentence Embeddings for Zero-Shot Cross-Lingual Transfer and Beyondcited by 434
  2. 2022DeepNet: Scaling Transformers to 1,000 Layerscited by 178
  3. 2019CCMatrix: Mining Billions of High-Quality Parallel Sentences on the WEBcited by 177
  4. 2021Direct speech-to-speech translation with discrete unitscited by 149
  5. 2017Sequence-to-Sequence Models Can Directly Translate Foreign Speechcited by 121
  6. 2021Textless Speech-to-Speech Translation on Real Datacited by 121
  7. 2019Direct speech-to-speech translation with a sequence-to-sequence modelcited by 118
  8. 2016Listen and Translate: A Proof of Concept for End-to-End Speech-to-Text Translationcited by 113
  9. 2022mSLAM: Massively multilingual joint pre-training for speech and textcited by 108
  10. 2021The Multilingual TEDx Corpus for Speech Recognition and Translationcited by 101
  11. 2020CoVoST 2 and Massively Multilingual Speech-to-Text Translationcited by 74
  12. 2019The Missing Ingredient in Zero-Shot Neural Machine Translationcited by 59
  13. 2020CoVoST: A Diverse Multilingual Speech-To-Text Translation Corpuscited by 57
  14. 2020Bridging the Modality Gap for Speech-to-Text Translationcited by 55
  15. 2021Stacked Acoustic-and-Textual Encoding: Integrating the Pre-trained Models into Speech Translation Encoderscited by 48
  16. 2019Improving Zero-shot Translation with Language-Independent Constraintscited by 33
  17. 2018A neural interlingua for multilingual machine translationcited by 29
  18. 2020Multilingual Machine Translation: Closing the Gap between Shared and Language-specific Encoder-Decoderscited by 23
  19. 2018Multilingual NMT with a language-independent attention bridgecited by 20
  20. 2017Towards speech-to-text translation without speech recognitioncited by 11
  21. 2020Enabling Zero-shot Multilingual Spoken Language Translation with Language-Specific Encoders and Decoderscited by 10
  22. 2020Training Multilingual Machine Translation by Alternately Freezing Language-Specific Encoders-Decoderscited by 8
  23. 2019Towards Interlingua Neural Machine Translationcited by 5
  24. 2021Zero-shot Speech Translationcited by 5
  25. 2022Tackling data scarcity in speech translation using zero-shot multilingual machine translation techniquescited by 4
  26. 2021Improving Zero-shot Neural Machine Translation on Language-specific Encoders-Decoderscited by 3
  27. 2022Triangular Transfer: Freezing the Pivot for Triangular Machine Translationcited by 2

Built on by

Cited by year

Citing papers per year: 2022: 2; 2023: 6; 2024: 6; 2025: 2.

16 papers.

Sort:

By area:

  1. 2023SONAR: Sentence-Level Multimodal and Language-Agnostic Representationscited by 108
  2. 2024Large Concept Models: Language Modeling in a Sentence Representation Spacecited by 89
  3. 2023Prompting the Hidden Talent of Web-Scale Speech Models for Zero-Shot Task Generalizationcited by 43
  4. 2022SpeechMatrix: A Large-Scale Mined Corpus of Multilingual Speech-to-Speech Translationscited by 35
  5. 2025Middle-Layer Representation Alignment for Cross-Lingual Transfer in Fine-Tuned LLMscited by 29
  6. 2022Speech-to-Speech Translation For A Real-world Unwritten Languagecited by 25
  7. 2023CMOT: Cross-modal Mixup via Optimal Transport for Speech Translationcited by 22
  8. 2024Pushing the Limits of Zero-shot End-to-End Speech Translationcited by 14
  9. 2024Evaluation of pseudo-healthy image reconstruction for anomaly detection with deep generative models: Application to brain FDG PETcited by 10
  10. 2024Can We Achieve High-quality Direct Speech-to-Speech Translation without Parallel Speech Data?cited by 4
  11. 2024CoSTA: Code-Switched Speech Translation using Aligned Speech-Text Interleavingcited by 4
  12. 2023Textless Speech-to-Speech Translation With Limited Parallel Datacited by 3
4 more
  1. 2025Speech-to-Text Translation with Phoneme-Augmented CoT: Enhancing Cross-Lingual Transfer in Low-Resource Scenarioscited by 3
  2. 2023Modular Speech-to-Text Translation for Zero-Shot Cross-Modal Transfercited by 2
  3. 2024Making Sentence Embeddings Robust to User-Generated Contentcited by 1
  4. 2023Direct Text to Speech Translation System using Acoustic Unitscited by 0