mlsanityData to 14 Oct 2026

Recent Advances in Direct Speech-to-text Translation

arXiv 2306.11646 ↗First posted 20 Jun 2023cs.CL eess.AScited by 15

Not on the map. Why?

Abstract (arXiv)

Recently, speech-to-text translation has attracted more and more attention and many studies have emerged rapidly. In this paper, we present a comprehensive survey on direct speech translation aiming to summarize the current state-of-the-art techniques. First, we categorize the existing research work into three directions based on the main challenges -- modeling burden, data scarcity, and application issues. To tackle the problem of modeling burden, two main structures have been proposed, encoder-decoder framework (Transformer and the variants) and multitask frameworks. For the challenge of data scarcity, recent work resorts to many sophisticated techniques, such as data augmentation, pre-training, knowledge distillation, and multilingual modeling. We analyze and summarize the application issues, which include real-time, segmentation, named entity, gender bias, and code-switching. Finally, we discuss some promising directions for future work.

Key facts. Cited by 15 papers in the map (data through 14 Oct 2026). Builds on 60 of its 77 references that are in the map. First posted on arXiv 20 Jun 2023.

Builds on

60 of 77 references.

References · 60

  1. 2017Attention Is All You Needcited by 60,361
  2. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,905
  3. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,342
  4. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,984
  5. 2020Conformer: Convolution-augmented Transformer for Speech Recognitioncited by 1,863
  6. 2019SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognitioncited by 1,380
  7. 2020Multilingual Denoising Pre-training for Neural Machine Translationcited by 1,035
  8. 2022data2vec: A General Framework for Self-supervised Learning in Speech, Vision and Languagecited by 727
  9. 2020Unsupervised Cross-lingual Representation Learning for Speech Recognitioncited by 501
  10. 2021SpeechT5: Unified-Modal Encoder-Decoder Pre-Training for Spoken Language Processingcited by 180
  11. 2017Sequence-to-Sequence Models Can Directly Translate Foreign Speechcited by 121
  12. 2016Listen and Translate: A Proof of Concept for End-to-End Speech-to-Text Translationcited by 113
38 more
  1. 2022mSLAM: Massively multilingual joint pre-training for speech and textcited by 108
  2. 2020Speech Translation and the End-to-End Promise: Taking Stock of Where We Arecited by 77
  3. 2022Unified Speech-Text Pre-training for Speech Translation and Recognitioncited by 64
  4. 2022Cross-modal Contrastive Learning for Speech Translationcited by 61
  5. 2019End-to-End Speech Translation with Knowledge Distillationcited by 58
  6. 2018Leveraging Weakly Supervised Data to Improve End-to-End Speech-to-Text Translationcited by 57
  7. 2020CoVoST: A Diverse Multilingual Speech-To-Text Translation Corpuscited by 57
  8. 2020Curriculum Pre-training for End-to-End Speech Translationcited by 57
  9. 2020Gender in Danger? Evaluating Speech Translation Technology on the MuST-SHE Corpuscited by 57
  10. 2022STEMM: Self-learning with Speech-text Manifold Mixup for Speech Translationcited by 57
  11. 2020Bridging the Modality Gap for Speech-to-Text Translationcited by 55
  12. 2018Tied Multitask Learning for Neural Speech Translationcited by 54
  13. 2020Dual-decoder Transformer for Joint Automatic Speech Recognition and Multilingual Speech Translationcited by 54
  14. 2021Cascade versus Direct Speech Translation: Do the Differences Still Make a Difference?cited by 54
  15. 2021Lightweight Adapter Tuning for Multilingual Speech Translationcited by 54
  16. 2021Improving Speech Translation by Understanding and Learning from the Auxiliary Text Translation Taskcited by 54
  17. 2020A General Multi-Task Learning Framework to Leverage Text Data for Speech to Text Taskscited by 51
  18. 2021Learning Shared Semantic Space for Speech-to-Text Translationcited by 49
  19. 2021Stacked Acoustic-and-Textual Encoding: Integrating the Pre-trained Models into Speech Translation Encoderscited by 48
  20. 2021Fused Acoustic and Text Encoding for Multimodal Bilingual Pretraining and Speech Translationcited by 47
  21. 2021End-to-end Speech Translation via Cross-modal Progressive Trainingcited by 47
  22. 2019Multilingual End-to-End Speech Translationcited by 37
  23. 2020Self-Training for End-to-End Speech Translationcited by 36
  24. 2021Large-Scale Self- and Semi-Supervised Learning for Speech Translationcited by 36
  25. 2020End-to-End Speech-Translation with Knowledge Distillation: FBK@IWSLT2020cited by 31
  26. 2019Attention-Passing Models for Robust and Data-Efficient End-to-End Speech Translationcited by 30
  27. 2021Source and Target Bidirectional Knowledge Distillation for End-to-end Speech Translationcited by 28
  28. 2022Sample, Translate, Recombine: Leveraging Audio Alignments for Data Augmentation in End-to-end Speech Translationcited by 27
  29. 2022SHAS: Approaching optimal Segmentation for End-to-End Speech Translationcited by 26
  30. 2022End-to-End Speech Translation for Code Switched Speechcited by 26
  31. 2022Wav2Seq: Pre-training Speech-to-Text Encoder-Decoder Models Using Pseudo Languagescited by 26
  32. 2021Investigating the Reordering Capability in CTC-based Non-Autoregressive End-to-End Speech Translationcited by 23
  33. 2022Under the Morphosyntactic Lens: A Multifaceted Evaluation of Gender Bias in Speech Translationcited by 23
  34. 2019Synchronous Speech Recognition and Speech-to-Text Translation with Interactive Decodingcited by 21
  35. 2020Self-Supervised Representations Improve End-to-End Speech Translationcited by 21
  36. 2019On Using SpecAugment for End-to-End Speech Translationcited by 19
  37. 2020Orthros: Non-autoregressive End-to-end Speech Translation with Dual-decodercited by 17
  38. 2021Beyond Voice Activity Detection: Hybrid Audio Segmentation for Direct Speech Translationcited by 17

50 most cited of 60 All 60 (JSON)

Built on by

Cited by year

Citing papers per year: 2023: 3; 2024: 7; 2025: 5.

15 papers.

Sort:

  1. 2024GenTranslate: Large Language Models are Generative Multilingual Speech and Machine Translatorscited by 22
  2. 2024LLaST: Improved End-to-end Speech Translation System Leveraged by Large Language Modelscited by 12
  3. 2023End-to-End Speech-to-Text Translation: A Surveycited by 9
  4. 2024Contrastive Learning for Task-Independent SpeechLLM-Pretrainingcited by 8
  5. 2025Hearing to Translate: The Effectiveness of Speech Modality Integration into LLMscited by 7
  6. 2023Rethinking and Improving Multi-task Learning for End-to-end Speech Translationcited by 6
  7. 2025Low-Rank and Sparse Model Merging for Multi-Lingual Speech Recognition and Translationcited by 6
  8. 2024Recent Advances in End-to-End Simultaneous Speech Translationcited by 5
  9. 2025Voice, Bias, and Coreference: An Interpretability Study of Gender in Speech Translationcited by 5
  10. 2024SBAAM! Eliminating Transcript Dependency in Automatic Subtitlingcited by 1
  11. 2024A Modular-based Strategy for Mitigating Gradient Conflicts in Simultaneous Speech Translationcited by 1
  12. 2023Bridging the Gaps of Both Modality and Language: Synchronous Bilingual CTC for Speech Translation and Speech Recognitioncited by 0
3 more
  1. 2024Revisiting Interpolation Augmentation for Speech-to-Text Generationcited by 0
  2. 2025Speech-to-Speech Translation Pipelines for Conversations in Low-Resource Languagescited by 0
  3. 2025HITSZ's End-To-End Speech Translation Systems Combining Sequence-to-Sequence Auto Speech Recognition Model and Indic Large Language Model for IWSLT 2025 in Indic Trackcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023Improved Cross-Lingual Transfer Learning For Automatic Speech Translation
  2. 2025Improving Language and Modality Transfer in Translation by Character-level Modeling
  3. 2025Bridging the Linguistic Divide: A Survey on Leveraging Large Language Models for Machine Translation
  4. 2025Revisiting Direct Speech-to-Text Translation with Speech LLMs: Better Scaling than CoT Prompting?
  5. 2024Investigating Decoder-only Large Language Models for Speech-to-text Translation
  6. 2023Decouple Non-parametric Knowledge Distillation For End-to-end Speech Translation
  7. 2025Towards Fine-Grained Code-Switch Speech Translation with Semantic Space Alignment
  8. 2022SpeechMatrix: A Large-Scale Mined Corpus of Multilingual Speech-to-Speech Translations