mlsanityData to 7 Oct 2026

ESPnet-ST-v2: Multipurpose Spoken Language Translation Toolkit

arXiv 2304.04596 ↗First posted 10 Apr 2023cs.SD cs.CL eess.AScited by 10

Not on the map. Why?

Abstract (arXiv)

ESPnet-ST-v2 is a revamp of the open-source ESPnet-ST toolkit necessitated by the broadening interests of the spoken language translation community. ESPnet-ST-v2 supports 1) offline speech-to-text translation (ST), 2) simultaneous speech-to-text translation (SST), and 3) offline speech-to-speech translation (S2ST) -- each task is supported with a wide variety of approaches, differentiating ESPnet-ST-v2 from other open source spoken language translation toolkits. This toolkit offers state-of-the-art architectures such as transducers, hybrid CTC/attention, multi-decoders with searchable intermediates, time-synchronous blockwise CTC/attention, Translatotron models, and direct discrete unit models. In this paper, we describe the overall design, example models for each task, and performance benchmarking behind ESPnet-ST-v2, which is publicly available at https://github.com/espnet/espnet.

Key facts. Cited by 10 papers in the map (data through 7 Oct 2026). Builds on 45 of its 64 references that are in the map. First posted on arXiv 10 Apr 2023.

Builds on

45 of 64 references.

References · 45

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020Conformer: Convolution-augmented Transformer for Speech Recognitioncited by 1,838
  3. 2018A Call for Clarity in Reporting BLEU Scorescited by 1,790
  4. 2019fairseq: A Fast, Extensible Toolkit for Sequence Modelingcited by 1,486
  5. 2020HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesiscited by 1,139
  6. 2020Multilingual Denoising Pre-training for Neural Machine Translationcited by 1,028
  7. 2021SUPERB: Speech processing Universal PERformance Benchmarkcited by 652
  8. 2017Tacotron: Towards End-to-End Speech Synthesiscited by 487
  9. 2021Speech Resynthesis from Discrete Disentangled Self-Supervised Representationscited by 243
  10. 2019NeMo: a toolkit for building AI applications using Neural Modulescited by 230
  11. 2019A Comparative Study on Transformer vs RNN in Speech Applicationscited by 210
  12. 2020fairseq S2T: Fast Speech-to-Text Modeling with fairseqcited by 174
33 more
  1. 2021Direct speech-to-speech translation with discrete unitscited by 148
  2. 2020Recent Developments on ESPnet Toolkit Boosted by Conformercited by 132
  3. 2021Textless Speech-to-Speech Translation on Real Datacited by 121
  4. 2019Direct speech-to-speech translation with a sequence-to-sequence modelcited by 118
  5. 2021Intermediate Loss Regularization for CTC-based Speech Recognitioncited by 95
  6. 2020SimulEval: An Evaluation Toolkit for Simultaneous Translationcited by 94
  7. 2022Branchformer: Parallel MLP-Attention Architectures to Capture Local and Global Context for Speech Recognition and Understandingcited by 90
  8. 2022E-Branchformer: Branchformer with Enhanced merging for speech recognitioncited by 90
  9. 2019Lingvo: a Modular and Scalable Framework for Sequence-to-Sequence Modelingcited by 86
  10. 2021TorchAudio: Building Blocks for Audio and Speech Processingcited by 83
  11. 2020ESPnet-ST: All-in-One Speech Translation Toolkitcited by 82
  12. 2020SimulMT to SimulST: Adapting Simultaneous Text Translation to End-to-End Simultaneous Speech Translationcited by 80
  13. 2022CVSS Corpus and Massively Multilingual Speech-to-Speech Translationcited by 71
  14. 2021Translatotron 2: High-quality direct speech-to-speech translation with voice preservationcited by 69
  15. 2019ESPnet-TTS: Unified, Reproducible, and Integratable Open Source End-to-End Text-to-Speech Toolkitcited by 60
  16. 2022UnitY: Two-pass Direct Speech-to-speech Translation with Discrete Unitscited by 60
  17. 2021Cascade versus Direct Speech Translation: Do the Differences Still Make a Difference?cited by 54
  18. 2022Quality-Aware Decoding for Neural Machine Translationcited by 49
  19. 2020Low-Latency Sequence-to-Sequence Speech Recognition and Translation by Partial Hypothesis Selectioncited by 47
  20. 2021CTC-based Compression for Direct Speech Translationcited by 45
  21. 2020UWSpeech: Speech to Speech Translation for Unwritten Languagescited by 43
  22. 2021Direct Simultaneous Speech-to-Text Translation Assisted by Synchronized Streaming ASRcited by 31
  23. 2022Large-Scale Streaming End-to-End Speech Translation with Neural Transducerscited by 31
  24. 2022CUNI-KIT System for Simultaneous Speech Translation Task at IWSLT 2022cited by 30
  25. 2021Source and Target Bidirectional Knowledge Distillation for End-to-end Speech Translationcited by 27
  26. 2022CTC Alignments Improve Autoregressive Translationcited by 25
  27. 2022Speech-to-Speech Translation For A Real-world Unwritten Languagecited by 25
  28. 2021Searchable Hidden Intermediates for End-to-End Models of Decomposable Sequence Taskscited by 24
  29. 2020NeurST: Neural Speech Translation Toolkitcited by 23
  30. 2021ESPnet2-TTS: Extending the Edge of TTS Researchcited by 22
  31. 2018Hierarchical Multi Task Learning With CTCcited by 20
  32. 2022Combining Spectral and Self-Supervised Features for Low Resource Speech Recognition and Translationcited by 11
  33. 2022Blockwise Streaming Transformer for Spoken Language Understanding and Simultaneous Speech Translationcited by 11

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 5; 2024: 3; 2025: 1; 2026: 1.

10 papers.

  1. 2023Exploring Speech Recognition, Translation, and Understanding with Discrete Speech Units: A Comparative Studycited by 42
  2. 2023Long-Form End-to-End Speech Translation via Latent Alignment Segmentationcited by 6
  3. 2024Label-Synchronous Neural Transducer for E2E Simultaneous Speech Translationcited by 5
  4. 2023Enhancing End-to-End Conversational Speech Translation Through Target Language Context Utilizationcited by 3
  5. 2023Cross-Modal Multi-Tasking for Speech-to-Text Translation via Hard Parameter Sharingcited by 3
  6. 2023Long-form Simultaneous Speech Translation: Thesis Proposalcited by 3
  7. 2024Lightweight Model Attribution and Detection of Synthetic Speech via Audio Residual Fingerprintscited by 3
  8. 2025HENT-SRT: Hierarchical Efficient Neural Transducer with Self-Distillation for Joint Speech Recognition and Translationcited by 1
  9. 2024ESPnet-EZ: Python-only ESPnet for Easy Fine-tuning and Integrationcited by 0
  10. 2026The Costs of Reproducibility in Music Separation Research: a Replication of Band-Split RNNcited by 0