mlsanityData to 14 Oct 2026

ESPnet-EZ: Python-only ESPnet for Easy Fine-tuning and Integration

arXiv 2409.09506 ↗First posted 14 Sep 2024cs.SD cs.AI eess.AScited by 0

Not on the map. Why?

Abstract (arXiv)

We introduce ESPnet-EZ, an extension of the open-source speech processing toolkit ESPnet, aimed at quick and easy development of speech models. ESPnet-EZ focuses on two major aspects: (i) easy fine-tuning and inference of existing ESPnet models on various tasks and (ii) easy integration with popular deep neural network frameworks such as PyTorch-Lightning, Hugging Face transformers and datasets, and Lhotse. By replacing ESPnet design choices inherited from Kaldi with a Python-only, Bash-free interface, we dramatically reduce the effort required to build, debug, and use a new model. For example, to fine-tune a speech foundation model, ESPnet-EZ, compared to ESPnet, reduces the number of newly written code by 2.7x and the amount of dependent code by 6.7x while dramatically reducing the Bash script dependencies. The codebase of ESPnet-EZ is publicly available.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 50 of its 65 references that are in the map. First posted on arXiv 14 Sep 2024.

Builds on

50 of 65 references.

References · 50

  1. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
  2. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  3. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  4. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 4,057
  5. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,342
  6. 2018A Call for Clarity in Reporting BLEU Scorescited by 1,799
  7. 2021WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processingcited by 1,657
  8. 2018VoxCeleb2: Deep Speaker Recognitioncited by 875
  9. 2017VoxCeleb: a large-scale speaker identification datasetcited by 732
  10. 2021SUPERB: Speech processing Universal PERformance Benchmarkcited by 666
  11. 2019LibriTTS: A Corpus Derived from LibriSpeech for Text-to-Speechcited by 649
  12. 2018ESPnet: End-to-End Speech Processing Toolkitcited by 558
38 more
  1. 2021Datasets: A Community Library for Natural Language Processingcited by 530
  2. 2021SpeechBrain: A General-Purpose Speech Toolkitcited by 406
  3. 2022UTMOS: UTokyo-SaruLab System for VoiceMOS Challenge 2022cited by 390
  4. 2019NeMo: a toolkit for building AI applications using Neural Modulescited by 235
  5. 2020SLURP: A Spoken Language Understanding Resource Packagecited by 198
  6. 2021Unsupervised Speech Recognitioncited by 148
  7. 2021WeNet: Production oriented Streaming and Non-streaming End-to-End Speech Recognition Toolkitcited by 125
  8. 2021TorchAudio: Building Blocks for Audio and Speech Processingcited by 83
  9. 2020ESPnet-ST: All-in-One Speech Translation Toolkitcited by 82
  10. 2020Voice Conversion Challenge 2020: Intra-lingual semi-parallel and cross-lingual voice conversioncited by 75
  11. 2024OWSM v3.1: Better and Faster Open Whisper-Style Speech Models based on E-Branchformercited by 74
  12. 2021SLUE: New Benchmark Tasks for Spoken Language Understanding Evaluation on Natural Speechcited by 72
  13. 2023Voxtlm: unified decoder-only models for consolidating speech recognition/synthesis and speech/text continuation taskscited by 72
  14. 2023ML-SUPERB: Multilingual Speech Universal PERformance Benchmarkcited by 71
  15. 2019ESPnet-TTS: Unified, Reproducible, and Integratable Open Source End-to-End Text-to-Speech Toolkitcited by 60
  16. 2021ESPnet-SLU: Advancing Spoken Language Understanding through ESPnetcited by 53
  17. 2020Asteroid: the PyTorch-based audio source separation toolkit for researcherscited by 45
  18. 2023Prompting the Hidden Talent of Web-Scale Speech Models for Zero-Shot Task Generalizationcited by 43
  19. 2023Reproducing Whisper-Style Training Using an Open-Source Toolkit and Publicly Available Datacited by 42
  20. 2024OWSM-CTC: An Open Encoder-Only Speech Foundation Model for Speech Recognition, Translation, and Language Identificationcited by 37
  21. 2021Lhotse: a speech data representation library for the modern deep learning ecosystemcited by 33
  22. 2024ML-SUPERB 2.0: Benchmarking Multilingual Speech Models Across Modeling Constraints, Languages, and Datasetscited by 27
  23. 2022CTC Alignments Improve Autoregressive Translationcited by 25
  24. 2020ESPnet-se: end-to-end speech enhancement and separation toolkit designed for asr integrationcited by 22
  25. 2021ESPnet2-TTS: Extending the Edge of TTS Researchcited by 22
  26. 2022Evaluate & Evaluation on the Hub: Better Best Practices for Data and Model Measurementscited by 22
  27. 2023The CHiME-7 DASR Challenge: Distant Meeting Transcription with Multiple Devices in Diverse Scenarioscited by 22
  28. 2024ESPnet-SPK: full pipeline speaker embedding toolkit with reproducible recipes, self-supervised front-ends, and off-the-shelf modelscited by 22
  29. 2023Reducing Barriers to Self-Supervised Learning: HuBERT Pre-training with Academic Computecited by 21
  30. 2021Temporal Knowledge Distillation for On-device Audio Classificationcited by 15
  31. 2022Muskits: an End-to-End Music Processing Toolkit for Singing Voice Synthesiscited by 10
  32. 2023ESPnet-ST-v2: Multipurpose Spoken Language Translation Toolkitcited by 10
  33. 2023Findings of the 2023 ML-SUPERB Challenge: Pre-Training and Evaluation over More Languages and Beyondcited by 10
  34. 2024On the Effects of Heterogeneous Data Sources on Speech-to-Text Foundation Modelscited by 10
  35. 2022ESPnet-SE++: Speech Enhancement for Robust Speech Recognition, Translation, and Understandingcited by 8
  36. 2024Wav2Gloss: Generating Interlinear Glossed Text from Speechcited by 8
  37. 2024USAT: A Universal Speaker-Adaptive Text-to-Speech Approachcited by 6
  38. 2022EURO: ESPnet Unsupervised ASR Open-source Toolkitcited by 5

Built on by

Not cited yet.