mlsanityData to 7 Oct 2026

An Efficient Self-Learning Framework For Interactive Spoken Dialog Systems

arXiv 2409.10515 ↗First posted 16 Sep 2024eess.AS cs.AI cs.CLcited by 0

Not on the map. Why?

Abstract (arXiv)

Dialog systems, such as voice assistants, are expected to engage with users in complex, evolving conversations. Unfortunately, traditional automatic speech recognition (ASR) systems deployed in such applications are usually trained to recognize each turn independently and lack the ability to adapt to the conversational context or incorporate user feedback. In this work, we introduce a general framework for ASR in dialog systems that can go beyond learning from single-turn utterances and learn over time how to adapt to both explicit supervision and implicit user feedback present in multi-turn conversations. We accomplish that by leveraging advances in student-teacher learning and context-aware dialog processing, and designing contrastive self-supervision approaches with Ohm, a new online hard-negative mining approach. We show that leveraging our new framework compared to traditional training leads to relative WER reductions of close to 10% in real-world dialog systems, and up to 26% on public synthetic data.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 47 of its 70 references that are in the map. First posted on arXiv 16 Sep 2024.

Builds on

47 of 70 references.

References · 47

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  3. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 3,972
  4. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,853
  5. 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,973
  6. 2020Conformer: Convolution-augmented Transformer for Speech Recognitioncited by 1,838
  7. 2020Contrastive Learning with Hard Negative Samplescited by 619
  8. 2019Be Your Own Teacher: Improve the Performance of Convolutional Neural Networks via Self Distillationcited by 491
  9. 2021Combined Scaling for Zero-shot Transfer Learningcited by 182
  10. 2017Data Distillation: Towards Omni-Supervised Learningcited by 126
  11. 2023FLIRT: Feedback Loop In-context Red Teamingcited by 81
  12. 2023Generative Speech Recognition Error Correction with Large Language Models and Task-Activating Promptingcited by 71
35 more
  1. 2022Contextual Adapters for Personalized Speech Recognition in Neural Transducerscited by 57
  2. 2021Context-Aware Transformer Transducer for Speech Recognitioncited by 56
  3. 2019Personalizing ASR for Dysarthric and Accented Speech with Limited Datacited by 45
  4. 2018Dialog-context aware end-to-end speech recognitioncited by 27
  5. 2019Lessons from Building Acoustic Models with a Million Hours of Speechcited by 27
  6. 2021Personalization Strategies for End-to-End Speech Recognition Systemscited by 26
  7. 2018Personalized Language Model for Query Auto-Completioncited by 25
  8. 2021Advanced Long-context End-to-end Speech Recognition Using Context-expanded Transformerscited by 25
  9. 2021Fast Contextual Adaptation with Neural Associative Memory for On-Device Personalized Speech Recognitioncited by 25
  10. 2022WAVPROMPT: Towards Few-Shot Spoken Language Understanding with Frozen Language Modelscited by 25
  11. 2022Revisiting Self-Distillationcited by 23
  12. 2023From English to More Languages: Parameter-Efficient Model Reprogramming for Cross-Lingual Speech Recognitioncited by 23
  13. 2022SpeechPrompt: An Exploration of Prompt Tuning on Generative Spoken Language Model for Speech Processing Taskscited by 19
  14. 2022Low-Resource Music Genre Classification with Cross-Modal Neural Model Reprogrammingcited by 17
  15. 2024Improving ASR Contextual Biasing with Guided Attentioncited by 17
  16. 2019Gated Embeddings in End-to-End Speech Recognition for Conversational-Context Fusioncited by 14
  17. 2021Hierarchical Transformer-based Large-Context End-to-end ASR with Large-Context Knowledge Distillationcited by 12
  18. 2021Multi-Modal Pre-Training for Automated Speech Recognitioncited by 10
  19. 2018Sequence-Level Knowledge Distillation for Model Compression of Attention-based Sequence-to-Sequence Speech Recognitioncited by 9
  20. 2022A Scalable Model Specialization Framework for Training and Inference using Submodels and its Application to Speech Model Personalizationcited by 8
  21. 2022Content-Context Factorized Representations for Automated Speech Recognitioncited by 8
  22. 2021Configurable Privacy-Preserving Automatic Speech Recognitioncited by 7
  23. 2023Contextual Biasing of Named-Entities with Large Language Modelscited by 7
  24. 2023Retrieve and Copy: Scaling ASR Personalization to Large Catalogscited by 7
  25. 2017Dialog Context Language Modeling with Recurrent Neural Networkscited by 6
  26. 2022Distilling the Knowledge of BERT for CTC-based ASRcited by 6
  27. 2023Personalized Predictive ASR for Latency Reduction in Voice Assistantscited by 6
  28. 2021Attentive Contextual Carryover for Multi-Turn End-to-End Spoken Language Understandingcited by 5
  29. 2022CUE Vectors: Modular Training of Language Models Conditioned on Diverse Contextual Signalscited by 5
  30. 2021Contextual Biasing of Language Models for Speech Recognition in Goal-Oriented Conversational Agentscited by 4
  31. 2022ILASR: Privacy-Preserving Incremental Learning for Automatic Speech Recognition at Production Scalecited by 4
  32. 2022Unified Modeling of Multi-Domain Multi-Device ASR Systemscited by 3
  33. 2024Promptformer: Prompted Conformer Transducer for ASRcited by 3
  34. 2024Task Oriented Dialogue as a Catalyst for Self-Supervised Automatic Speech Recognitioncited by 1
  35. 2024Multi-Stage Multi-Modal Pre-Training for Automatic Speech Recognitioncited by 1

Built on by

Not cited yet.