mlsanityData to 7 Oct 2026

Cross-modal Contrastive Learning for Speech Translation

arXiv 2205.02444 ↗First posted 5 May 2022cs.CL eess.AScited by 61

Not on the map. Why?

Abstract (arXiv)

How can we learn unified representations for spoken utterances and their written text? Learning similar representations for semantically similar speech and text is important for speech translation. To this end, we propose ConST, a cross-modal contrastive learning method for end-to-end speech-to-text translation. We evaluate ConST and a variety of previous baselines on a popular benchmark MuST-C. Experiments show that the proposed ConST consistently outperforms the previous methods on, and achieves an average BLEU of 29.4. The analysis further verifies that ConST indeed closes the representation gap of different modalities -- its learned representation improves the accuracy of cross-modal speech-text retrieval from 4% to 88%. Code and models are available at https://github.com/ReneeYe/ConST.

Key facts. Cited by 61 papers in the map (data through 7 Oct 2026). Builds on 60 of its 91 references that are in the map. First posted on arXiv 5 May 2022.

Builds on

60 of 91 references.

References · 60

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  3. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  4. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,712
  5. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,853
  6. 2015FaceNet: A Unified Embedding for Face Recognition and Clusteringcited by 2,833
  7. 2020Supervised Contrastive Learningcited by 2,803
  8. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,219
  9. 2021SimCSE: Simple Contrastive Learning of Sentence Embeddingscited by 2,168
  10. 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,973
  11. 2018A Call for Clarity in Reporting BLEU Scorescited by 1,790
  12. 2019fairseq: A Fast, Extensible Toolkit for Sequence Modelingcited by 1,486
38 more
  1. 2020BLEURT: Learning Robust Metrics for Text Generationcited by 1,196
  2. 2019Cross-lingual Language Model Pretrainingcited by 1,088
  3. 2020Multilingual Denoising Pre-training for Neural Machine Translationcited by 1,028
  4. 2019Unified Language Model Pre-training for Natural Language Understanding and Generationcited by 668
  5. 2019Unified Vision-Language Pre-Training for Image Captioning and VQAcited by 494
  6. 2019A Comprehensive Overhaul of Feature Distillationcited by 341
  7. 2015LSTM: A Search Space Odysseycited by 315
  8. 2020Improving Massively Multilingual Neural Machine Translation and Zero-Shot Translationcited by 277
  9. 2020UNIMO: Towards Unified-Modal Understanding and Generation via Cross-Modal Contrastive Learningcited by 239
  10. 2021ConSERT: A Contrastive Framework for Self-Supervised Sentence Representation Transfercited by 231
  11. 2017Parallel WaveNet: Fast High-Fidelity Speech Synthesiscited by 214
  12. 2020fairseq S2T: Fast Speech-to-Text Modeling with fairseqcited by 174
  13. 2020CERT: Contrastive Self-supervised Learning for Language Understandingcited by 169
  14. 2017Sequence-to-Sequence Models Can Directly Translate Foreign Speechcited by 121
  15. 2021CLIFF: Contrastive Learning for Improving Faithfulness and Factuality in Abstractive Summarizationcited by 120
  16. 2016Listen and Translate: A Proof of Concept for End-to-End Speech-to-Text Translationcited by 113
  17. 2021Contrastive Learning for Many-to-many Multilingual Neural Machine Translationcited by 108
  18. 2020ESPnet-ST: All-in-One Speech Translation Toolkitcited by 82
  19. 2021ESimCSE: Enhanced Sample Building Method for Contrastive Learning of Unsupervised Sentence Embeddingcited by 73
  20. 2020A Simple but Tough-to-Beat Data Augmentation Approach for Natural Language Understanding and Generationcited by 72
  21. 2018Pre-training on high-resource speech recognition improves low-resource speech-to-text translationcited by 70
  22. 2018End-to-End Automatic Speech Translation of Audiobookscited by 65
  23. 2022Unified Speech-Text Pre-training for Speech Translation and Recognitioncited by 64
  24. 2020Curriculum Pre-training for End-to-End Speech Translationcited by 57
  25. 2022STEMM: Self-learning with Speech-text Manifold Mixup for Speech Translationcited by 57
  26. 2020Bridging the Modality Gap for Speech-to-Text Translationcited by 55
  27. 2020Dual-decoder Transformer for Joint Automatic Speech Recognition and Multilingual Speech Translationcited by 54
  28. 2021Cascade versus Direct Speech Translation: Do the Differences Still Make a Difference?cited by 54
  29. 2021Improving Speech Translation by Understanding and Learning from the Auxiliary Text Translation Taskcited by 54
  30. 2021Lightweight Adapter Tuning for Multilingual Speech Translationcited by 53
  31. 2020A General Multi-Task Learning Framework to Leverage Text Data for Speech to Text Taskscited by 51
  32. 2021Learning Shared Semantic Space for Speech-to-Text Translationcited by 49
  33. 2021Stacked Acoustic-and-Textual Encoding: Integrating the Pre-trained Models into Speech Translation Encoderscited by 48
  34. 2021Fused Acoustic and Text Encoding for Multimodal Bilingual Pretraining and Speech Translationcited by 47
  35. 2021End-to-end Speech Translation via Cross-modal Progressive Trainingcited by 47
  36. 2020Self-Training for End-to-End Speech Translationcited by 36
  37. 2021Large-Scale Self- and Semi-Supervised Learning for Speech Translationcited by 36
  38. 2020Consecutive Decoding for Speech-to-text Translationcited by 28

50 most cited of 60 All 60 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 10; 2023: 29; 2024: 8; 2025: 11; 2026: 3.

61 papers.

Sort:

  1. 2022SpeechUT: Bridging Speech and Text with Hidden-Unit for Encoder-Decoder Based Speech-Text Pre-trainingcited by 40
  2. 2023AlignAtt: Using Attention-based Audio-Translation Alignments as a Guide for Simultaneous Speech Translationcited by 35
  3. 2025Middle-Layer Representation Alignment for Cross-Lingual Transfer in Fine-Tuned LLMscited by 26
  4. 2023Pre-training for Speech Translation: CTC Meets Optimal Transportcited by 25
  5. 2023CMOT: Cross-modal Mixup via Optimal Transport for Speech Translationcited by 22
  6. 2023End-to-End Simultaneous Speech Translation with Differentiable Segmentationcited by 22
  7. 2022M3ST: Mix at Three Levels for Speech Translationcited by 21
  8. 2022WACO: Word-Aligned Contrastive Learning for Speech Translationcited by 21
  9. 2023DUB: Discrete Unit Back-translation for Speech Translationcited by 18
  10. 2023Understanding and Bridging the Modality Gap for Speech Translationcited by 15
  11. 2023Recent Advances in Direct Speech-to-text Translationcited by 15
  12. 2024Pushing the Limits of Zero-shot End-to-End Speech Translationcited by 14
38 more
  1. 2023CWCL: Cross-Modal Transfer with Continuously Weighted Contrastive Losscited by 12
  2. 2024LLaST: Improved End-to-end Speech Translation System Leveraged by Large Language Modelscited by 12
  3. 2023Code-Switching Text Generation and Injection in Mandarin-English ASRcited by 10
  4. 2022Improving End-to-end Speech Translation by Leveraging Auxiliary Speech and Text Datacited by 9
  5. 2023Back Translation for Speech-to-text Translation Without Transcriptscited by 9
  6. 2023ComSL: A Composite Speech-Language Model for End-to-End Speech-to-Text Translationcited by 9
  7. 2023CTC-based Non-autoregressive Speech Translationcited by 9
  8. 2023Tuning Large language model for End-to-end Speech Translationcited by 9
  9. 2023End-to-End Speech-to-Text Translation: A Surveycited by 9
  10. 2024A Contrastive Learning Approach to Mitigate Bias in Speech Modelscited by 9
  11. 2024Contrastive Learning for Task-Independent SpeechLLM-Pretrainingcited by 8
  12. 2023Improving Speech Translation by Cross-Modal Multi-Grained Contrastive Learningcited by 7
  13. 2022SegAugment: Maximizing the Utility of Speech Translation Data with Segmentation-based Augmentationscited by 6
  14. 2023Adapting Offline Speech Translation Models for Streaming with Future-Aware Distillation and Inferencecited by 6
  15. 2023Modality Adaption or Regularization? A Case Study on End-to-End Speech Translationcited by 6
  16. 2023An Empirical Study of Consistency Regularization for End-to-End Speech-to-Text Translationcited by 6
  17. 2023Rethinking and Improving Multi-task Learning for End-to-end Speech Translationcited by 6
  18. 2023ML-LMCL: Mutual Learning and Large-Margin Contrastive Learning for Improving ASR Robustness in Spoken Language Understandingcited by 6
  19. 2022Low-resource Neural Machine Translation with Cross-modal Alignmentcited by 5
  20. 2024Enhancing Multilingual Voice Toxicity Detection with Speech-Text Alignmentcited by 5
  21. 2022Learning Decoupled Retrieval Representation for Nearest Neighbour Neural Machine Translationcited by 4
  22. 2022Named Entity Detection and Injection for Direct Speech Translationcited by 4
  23. 2023Soft Alignment of Modality Space for End-to-end Speech Translationcited by 4
  24. 2024CoSTA: Code-Switched Speech Translation using Aligned Speech-Text Interleavingcited by 4
  25. 2022AdaTranS: Adapting with Boundary-based Shrinking for End-to-End Speech Translationcited by 3
  26. 2023Decouple Non-parametric Knowledge Distillation For End-to-end Speech Translationcited by 3
  27. 2023Improving speech translation by fusing speech and textcited by 3
  28. 2023Cross-Modal Multi-Tasking for Speech-to-Text Translation via Hard Parameter Sharingcited by 3
  29. 2023Towards a Deep Understanding of Multilingual End-to-End Speech Translationcited by 3
  30. 2024Representation Purification for End-to-End Speech Translationcited by 3
  31. 2025WhiSPA: Semantically and Psychologically Aligned Whisper with Self-Supervised Contrastive and Student-Teacher Learningcited by 3
  32. 2022Bootstrapping meaning through listening: Unsupervised learning of spoken sentence embeddingscited by 2
  33. 2023Self-Supervised Generative-Contrastive Learning of Multi-Modal Euclidean Input for 3D Shape Latent Representations: A Dynamic Switching Approachcited by 2
  34. 2023Modular Speech-to-Text Translation for Zero-Shot Cross-Modal Transfercited by 2
  35. 2024CLASP: Contrastive Language-Speech Pretraining for Multilingual Multimodal Information Retrievalcited by 2
  36. 2025Speech Translation Refinement using Large Language Modelscited by 2
  37. 2025Adaptive Inner Speech-Text Alignment for LLM-based Speech Translationcited by 2
  38. 2025DTW-Align: Bridging the Modality Gap in End-to-End Speech Translation with Dynamic Time Warping Alignmentcited by 2

50 most cited of 61 All 61 (JSON)