mlsanityData to 7 Oct 2026

Deep Contextualized Acoustic Representations For Semi-Supervised Speech Recognition

arXiv 1912.01679 ↗First posted 3 Dec 2019eess.AS cs.CL cs.LGcited by 66

Not on the map. Why?

Abstract (arXiv)

We propose a novel approach to semi-supervised automatic speech recognition (ASR). We first exploit a large amount of unlabeled audio data via representation learning, where we reconstruct a temporal slice of filterbank features from past and future context frames. The resulting deep contextualized acoustic representations (DeCoAR) are then used to train a CTC-based end-to-end ASR system using a smaller amount of labeled audio data. In our experiments, we show that systems trained on DeCoAR consistently outperform ones trained on conventional filterbank features, giving 42% and 19% relative improvement over the baseline on WSJ eval92 and LibriSpeech test-clean, respectively. Our approach can drastically reduce the amount of labeled data required; unsupervised training on LibriSpeech then supervision with 100 hours of labeled data achieves performance on par with training on all 960 hours directly. Pre-trained models and code will be released online.

Key facts. Cited by 66 papers in the map (data through 7 Oct 2026). First posted on arXiv 3 Dec 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 22; 2022: 20; 2023: 13; 2024: 7; 2025: 2; 2026: 2.

66 papers.

Sort:

By area:

  1. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,301
  2. 2021WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processingcited by 1,609
  3. 2022data2vec: A General Framework for Self-supervised Learning in Speech, Vision and Languagecited by 718
  4. 2021W2v-BERT: Combining Contrastive Learning and Masked Language Modeling for Self-Supervised Speech Pre-Trainingcited by 340
  5. 2021Generative Spoken Language Modeling from Raw Audiocited by 336
  6. 2021Layer-wise Analysis of a Self-supervised Speech Representation Modelcited by 293
  7. 2022Self-Supervised Speech Representation Learning: A Reviewcited by 258
  8. 2023Google USM: Scaling Automatic Speech Recognition Beyond 100 Languagescited by 228
  9. 2021DistilHuBERT: Speech Representation Learning by Layer-wise Distillation of Hidden-unit BERTcited by 129
  10. 2021BigSSL: Exploring the Frontier of Large-Scale Semi-Supervised Learning for Automatic Speech Recognitioncited by 88
  11. 2021UniSpeech: Unified Speech Representation Learning with Labeled and Unlabeled Datacited by 76
  12. 2022SUPERB-SG: Enhanced Speech processing Universal PERformance Benchmark for Semantic and Generative Capabilitiescited by 76
38 more
  1. 2021SLUE: New Benchmark Tasks for Spoken Language Understanding Evaluation on Natural Speechcited by 70
  2. 2022Audio Self-supervised Learning: A Surveycited by 65
  3. 2023Generative Pre-training for Speech with Flow Matchingcited by 55
  4. 2023MARBLE: Music Audio Representation Benchmark for Universal Evaluationcited by 38
  5. 2024A Large-Scale Evaluation of Speech Foundation Modelscited by 38
  6. 2021Momentum Pseudo-Labeling for Semi-Supervised Speech Recognitioncited by 37
  7. 2021SynthASR: Unlocking Synthetic Data for Speech Recognitioncited by 34
  8. 2022SUPERB @ SLT 2022: Challenge on Generalization and Efficiency of Self-Supervised Speech Representation Learningcited by 26
  9. 2021Wav2vec-C: A Self-supervised Model for Speech Representation Learningcited by 25
  10. 2023AV-data2vec: Self-supervised Learning of Audio-Visual Speech Representations with Contextualized Target Representationscited by 24
  11. 2021Scaling Laws for Acoustic Modelscited by 20
  12. 2021Don't speak too fast: The impact of data bias on self-supervised speech modelscited by 20
  13. 2023Accommodating Audio Modality in CLIP for Multimodal Processingcited by 18
  14. 2021Wav2vec-Switch: Contrastive Learning from Original-noisy Speech Pairs for Robust Speech Recognitioncited by 15
  15. 2021Contrastive Semi-supervised Learning for ASRcited by 11
  16. 2021Speech Technology for Everyone: Automatic Speech Recognition for Non-Native English with Transfer Learningcited by 9
  17. 2022A Brief Overview of Unsupervised Neural Speech Representation Learningcited by 9
  18. 2022MT4SSL: Boosting Self-Supervised Speech Representation Learning by Integrating Multiple Targetscited by 8
  19. 2023MiniSUPERB: Lightweight Benchmark for Self-supervised Speech Modelscited by 8
  20. 2022Silence is Sweeter Than Speech: Self-Supervised Model Using Silence to Store Speaker Informationcited by 7
  21. 2022Losses Can Be Blessings: Routing Self-Supervised Speech Representations Towards Efficient Multilingual and Multitask Speech Processingcited by 7
  22. 2023Pushing the Limits of Unsupervised Unit Discovery for SSL Speech Representationcited by 7
  23. 2022Avoid Overthinking in Self-Supervised Models for Speech Recognitioncited by 5
  24. 2024The Effect of Batch Size on Contrastive Self-Supervised Speech Representation Learningcited by 5
  25. 2022ILASR: Privacy-Preserving Incremental Learning for Automatic Speech Recognition at Production Scalecited by 4
  26. 2022Speech-text based multi-modal training with bidirectional attention for improved speech recognitioncited by 4
  27. 2021Fusion of Embeddings Networks for Robust Combination of Text Dependent and Independent Speaker Recognitioncited by 3
  28. 2021Improving Pseudo-label Training For End-to-end Speech Recognition Using Gradient Maskcited by 3
  29. 2022Improving Automatic Speech Recognition for Non-Native English with Transfer Learning and Language Model Decodingcited by 3
  30. 2023Self-supervised Neural Factor Analysis for Disentangling Utterance-level Speech Representationscited by 3
  31. 2024REBORN: Reinforcement-Learned Boundary Segmentation with Iterative Training for Unsupervised ASRcited by 3
  32. 2021Injecting Text and Cross-lingual Supervision in Few-shot Learning from Self-Supervised Modelscited by 2
  33. 2022Contrastive Siamese Network for Semi-supervised Speech Recognitioncited by 2
  34. 2022Improving Children's Speech Recognition by Fine-tuning Self-supervised Adult Speech Representationscited by 2
  35. 2023Sim-T: Simplify the Transformer Network by Multiplexing Technique for Speech Recognitioncited by 2
  36. 2023Don't Stop Self-Supervision: Accent Adaptation of Speech Representations via Residual Adapterscited by 2
  37. 2024On the Transferability of Large-Scale Self-Supervision to Few-Shot Audio Classificationcited by 2
  38. 2024Improving noisy student training for low-resource languages in End-to-End ASR using CycleGAN and inter-domain lossescited by 2

50 most cited of 66 All 66 (JSON)