mlsanityData to 14 Oct 2026

ESPnet-SLU: Advancing Spoken Language Understanding through ESPnet

arXiv 2111.14706 ↗First posted 29 Nov 2021cs.CL cs.SD eess.AScited by 53

Not on the map. Why?

Abstract (arXiv)

As Automatic Speech Processing (ASR) systems are getting better, there is an increasing interest of using the ASR output to do downstream Natural Language Processing (NLP) tasks. However, there are few open source toolkits that can be used to generate reproducible results on different Spoken Language Understanding (SLU) benchmarks. Hence, there is a need to build an open source standard that can be used to have a faster start into SLU research. We present ESPnet-SLU, which is designed for quick development of spoken language understanding in a single framework. ESPnet-SLU is a project inside end-to-end speech processing toolkit, ESPnet, which is a widely used open-source standard for various speech processing tasks like ASR, Text to Speech (TTS) and Speech Translation (ST). We enhance the toolkit to provide implementations for various SLU benchmarks that enable researchers to seamlessly mix-and-match different ASR and NLU models. We also provide pretrained models with intensively tuned hyper-parameters that can match or even outperform the current state-of-the-art performances. The toolkit is publicly available at https://github.com/espnet/espnet.

Key facts. Cited by 53 papers in the map (data through 14 Oct 2026). Builds on 40 of its 58 references that are in the map. First posted on arXiv 29 Nov 2021.

Builds on

40 of 58 references.

References · 40

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,905
  3. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,342
  4. 2020Conformer: Convolution-augmented Transformer for Speech Recognitioncited by 1,863
  5. 2019fairseq: A Fast, Extensible Toolkit for Sequence Modelingcited by 1,488
  6. 2019SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognitioncited by 1,380
  7. 2020MPNet: Masked and Permuted Pre-training for Language Understandingcited by 981
  8. 2018Speech Commands: A Dataset for Limited-Vocabulary Speech Recognitioncited by 752
  9. 2021SUPERB: Speech processing Universal PERformance Benchmarkcited by 666
  10. 2018ESPnet: End-to-End Speech Processing Toolkitcited by 558
  11. 2020Unsupervised Cross-lingual Representation Learning for Speech Recognitioncited by 501
  12. 2021SpeechBrain: A General-Purpose Speech Toolkitcited by 406
28 more
  1. 2018Snips Voice Platform: an embedded Spoken Language Understanding system for private-by-design voice interfacescited by 377
  2. 2019A Comparative Study on Transformer vs RNN in Speech Applicationscited by 211
  3. 2020SLURP: A Spoken Language Understanding Resource Packagecited by 198
  4. 2020TERA: Self-Supervised Learning of Transformer Encoder Representation for Speechcited by 188
  5. 2019WHAM!: Extending Speech Separation to Noisy Environmentscited by 156
  6. 2020ESPnet-ST: All-in-One Speech Translation Toolkitcited by 82
  7. 2020Vector-Quantized Autoregressive Predictive Codingcited by 69
  8. 2021Earnings-21: A Practical Benchmark for ASR in the Wildcited by 62
  9. 2019ESPnet-TTS: Unified, Reproducible, and Integratable Open Source End-to-End Text-to-Speech Toolkitcited by 60
  10. 2020Semi-Supervised Spoken Language Understanding via Self-Supervised Speech and Language Model Pretrainingcited by 35
  11. 2016Sequential Short-Text Classification with Recurrent and Convolutional Neural Networkscited by 28
  12. 2018Spoken Language Understanding on the Edgecited by 27
  13. 2020Tie Your Embeddings Down: Cross-Modal Latent Spaces for End-to-end Spoken Language Understandingcited by 23
  14. 2020ESPnet-se: end-to-end speech enhancement and separation toolkit designed for asr integrationcited by 22
  15. 2020Improving End-to-End Speech-to-Intent Classification with Reptilecited by 17
  16. 2021End-to-End Spoken Language Understanding for Generalized Voice Assistantscited by 17
  17. 2019Gated Embeddings in End-to-End Speech Recognition for Conversational-Context Fusioncited by 14
  18. 2019Cross-Attention End-to-End ASR for Two-Party Conversationscited by 14
  19. 2019Gunrock: A Social Bot for Complex and Engaging Long Conversationscited by 14
  20. 2020The Sequence-to-Sequence Baseline for the Voice Conversion Challenge 2020: Cascading ASR and TTScited by 12
  21. 2021Rethinking End-to-End Evaluation of Decomposable Tasks: A Case Study on Spoken Language Understandingcited by 11
  22. 2020HarperValleyBank: A Domain-Specific Spoken Dialog Corpuscited by 10
  23. 2021Integrating Dialog History into End-to-End Spoken Language Understanding Systemscited by 9
  24. 2018Lexico-acoustic Neural-based Models for Dialog Act Classificationcited by 8
  25. 2021Speak or Chat with Me: End-to-End Spoken Language Understanding System with Flexible Inputscited by 8
  26. 2017Neural-based Context Representation Learning for Dialog Act Classificationcited by 4
  27. 2019Context-aware Neural-based Dialog Act Classification on Automatically Generated Transcriptionscited by 3
  28. 2021Pre-training for low resource speech-to-intent applicationscited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 14; 2023: 24; 2024: 8; 2025: 4; 2026: 3.

53 papers.

Sort:

By area:

  1. 2022Branchformer: Parallel MLP-Attention Architectures to Capture Local and Global Context for Speech Recognition and Understandingcited by 91
  2. 2023Efficient Sequence Transduction by Jointly Predicting Tokens and Durationscited by 58
  3. 2023PromptTTS 2: Describing and Generating Voices with Text Promptcited by 48
  4. 2022SLUE Phase-2: A Benchmark Suite of Diverse Spoken Language Understanding Taskscited by 35
  5. 2022SAMU-XLSR: Semantically-Aligned Multimodal Utterance-level Cross-Lingual Speech Representationcited by 32
  6. 2023Structured Pruning of Self-Supervised Pre-trained Models for Speech Recognition and Understandingcited by 31
  7. 2023Can ChatGPT Detect Intent? Evaluating Large Language Models for Spoken Language Understandingcited by 21
  8. 2024Exploring the Capability of Mamba in Speech Applicationscited by 20
  9. 2022A Study on the Integration of Pre-trained SSL, ASR, LM and SLU Models for Spoken Language Understandingcited by 18
  10. 2023A Comparative Study on E-Branchformer vs Conformer in Speech Recognition, Translation, and Understanding Taskscited by 14
  11. 2023CWCL: Cross-Modal Transfer with Continuously Weighted Contrastive Losscited by 12
  12. 2022Blockwise Streaming Transformer for Spoken Language Understanding and Simultaneous Speech Translationcited by 11
38 more
  1. 2022BERT Meets CTC: New Formulation of End-to-End Speech Recognition with Pre-trained Masked Language Modelcited by 11
  2. 2022Token-level Sequence Labeling for Spoken Language Understanding using Compositional End-to-End Modelscited by 10
  3. 2023Findings of the 2023 ML-SUPERB Challenge: Pre-Training and Evaluation over More Languages and Beyondcited by 10
  4. 2023SpeechGLUE: How Well Can Self-Supervised Speech Models Capture Linguistic Knowledge?cited by 9
  5. 2023UniverSLU: Universal Spoken Language Understanding for Diverse Tasks with Natural Language Instructionscited by 9
  6. 2024Voice Attribute Editing with Text Promptcited by 9
  7. 2024On the Evaluation of Speech Foundation Models for Spoken Language Understandingcited by 9
  8. 2022Two-Pass Low Latency End-to-End Spoken Language Understandingcited by 8
  9. 2022ConceptBeam: Concept Driven Target Speech Extractioncited by 8
  10. 2022End-to-end Spoken Language Understanding with Tree-constrained Pointer Generatorcited by 6
  11. 2022Streaming Joint Speech Recognition and Disfluency Detectioncited by 6
  12. 2023Improving Textless Spoken Language Understanding with Discrete Units as Intermediate Targetcited by 6
  13. 2023A Study on the Integration of Pipeline and E2E SLU systems for Spoken Semantic Parsing toward STOP Quality Challengecited by 5
  14. 2025Chain-of-Thought Training for Open E2E Spoken Dialogue Systemscited by 5
  15. 2025Chain-of-Thought Reasoning in Streaming Full-Duplex End-to-End Spoken Dialogue Systemscited by 5
  16. 2026Optimizing Conversational Quality in Spoken Dialogue Systems with Reinforcement Learning from AI Feedbackcited by 5
  17. 2023The Interpreter Understands Your Meaning: End-to-end Spoken Language Understanding Aided by Speech Translationcited by 4
  18. 2022Bridging Speech and Textual Pre-trained Models with Unsupervised ASRcited by 3
  19. 2022Multitask Learning for Low Resource Spoken Language Understandingcited by 3
  20. 2023Joint Modelling of Spoken Language Understanding Tasks with Integrated Dialog Historycited by 3
  21. 2023Sequence-Level Knowledge Distillation for Class-Incremental End-to-End Spoken Language Understandingcited by 3
  22. 2023CIF-PT: Bridging Speech and Text Representations for Spoken Language Understanding via Continuous Integrate-and-Fire Pre-Trainingcited by 3
  23. 2023Knowledge-Aware Audio-Grounded Generative Slot Filling for Limited Annotated Datacited by 3
  24. 2023Leveraging Pretrained ASR Encoders for Effective and Efficient End-to-End Speech Intent Classification and Slot Fillingcited by 3
  25. 2023Integrating Pretrained ASR and LM to Perform Sequence Generation for Spoken Language Understandingcited by 2
  26. 2024Finding Task-specific Subnetworks in Multi-task Spoken Language Understanding Modelcited by 2
  27. 2024Enhancing Dialogue Speech Recognition with Robust Contextual Awareness via Noise Representation Learningcited by 2
  28. 2025Spoken Language Understanding on Unseen Tasks With In-Context Learningcited by 2
  29. 2023Leveraging Multilingual Self-Supervised Pretrained Models for Sequence-to-Sequence End-to-End Spoken Language Understandingcited by 1
  30. 2023Text generation for dataset augmentation in security classification taskscited by 1
  31. 2023Efficiency-oriented approaches for self-supervised speech representation learningcited by 1
  32. 2026TaigiSpeech: A Low-Resource Real-World Speech Intent Dataset and Preliminary Results with Scalable Data Mining In-the-Wildcited by 1
  33. 2026Selective Capability Unlearning in End-to-End Spoken Language Understandingcited by 1
  34. 2022Cross-stitched Multi-modal Encoderscited by 0
  35. 2023End-to-end spoken language understanding using joint CTC loss and self-supervised, pretrained acoustic encoderscited by 0
  36. 2023Continual Contrastive Spoken Language Understandingcited by 0
  37. 2023Improving End-to-End Speech Processing by Efficient Text Data Utilization with Latent Synthesiscited by 0
  38. 2024Privacy-Preserving End-to-End Spoken Language Understandingcited by 0

50 most cited of 53 All 53 (JSON)