mlsanityData to 19 Oct 2026

FAT-HuBERT: Front-end Adaptive Training of Hidden-unit BERT for Distortion-Invariant Robust Speech Recognition

arXiv 2311.17790 ↗First posted 29 Nov 2023cs.SD eess.AScited by 2

On the map: Speech and audio language models → Speech recognition models and training

Abstract (arXiv)

Advancements in monaural speech enhancement (SE) techniques have greatly improved the perceptual quality of speech. However, integrating these techniques into automatic speech recognition (ASR) systems has not yielded the expected performance gains, primarily due to the introduction of distortions during the SE process. In this paper, we propose a novel approach called FAT-HuBERT, which leverages distortion-invariant self-supervised learning (SSL) to enhance the robustness of ASR. To address the distortions introduced by the SE frontends, we introduce layer-wise fusion modules that incorporate features extracted from both observed noisy signals and enhanced signals. During training, the SE frontend is randomly selected from a pool of models. We evaluate the performance of FAT-HuBERT on simulated noisy speech generated from LibriSpeech as well as real-world noisy speech from the CHiME-4 1-channel dataset. The experimental results demonstrate a significant relative reduction in word error rate (WER).

Key facts. Cited by 2 papers in the map (data through 19 Oct 2026). Placed in Speech recognition models and training, one of 11 directions in Speech and audio language models. Builds on 31 of its 45 references that are in the map. First posted on arXiv 29 Nov 2023.

Builds on

31 of 45 references.

Closest ancestors · 6

Same direction, newest first

  1. 2022Robust Speech Recognition via Large-Scale Weak Supervision6 Dec
  2. 2022Contrastive Siamese Network for Semi-supervised Speech Recognition27 May
  3. 2022A Conformer Based Acoustic Model for Robust Automatic Speech Recognition1 Mar
  4. 2021Multi-Variant Consistency based Self-supervised Learning for Robust Automatic Speech Recognition23 Dec
  5. 2021Improving Noise Robustness of Contrastive Speech Representation Learning with Speech Reconstruction28 Oct
  6. 2021WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processing26 Oct

Other references · 25

  1. 2017Attention Is All You Needcited by 61,180
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,923
  3. 2018Representation Learning with Contrastive Predictive Codingcited by 7,970
  4. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 4,385
  5. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,762
  6. 2019fairseq: A Fast, Extensible Toolkit for Sequence Modelingcited by 1,533
  7. 2018ESPnet: End-to-End Speech Processing Toolkitcited by 754
  8. 2018Conv-TasNet: Surpassing Ideal Time-Frequency Magnitude Masking for Speech Separationcited by 596
  9. 2020DCCRN: Deep Complex Convolution Recurrent Network for Phase-Aware Speech Enhancementcited by 284
  10. 2019Dual-path RNN: efficient long sequence modeling for time-domain single-channel speech separationcited by 275
  11. 2019WHAM!: Extending Speech Separation to Noisy Environmentscited by 246
  12. 2020Multi-task self-supervised learning for Robust Speech Recognitioncited by 147
13 more
  1. 2019Phase-aware Speech Enhancement with Deep Complex U-Netcited by 110
  2. 2020Dual-Path Transformer Network: Direct Context-Aware Modeling for End-to-End Monaural Speech Separationcited by 105
  3. 2020DeCoAR 2.0: Deep Contextualized Acoustic Representations with Vector Quantizationcited by 84
  4. 2019Deep Contextualized Acoustic Representations For Semi-Supervised Speech Recognitioncited by 75
  5. 2022How Bad Are Artifacts?: Analyzing the Impact of Speech Enhancement Errors on ASRcited by 63
  6. 2017Deep Learning for Environmentally Robust Speech Recognition: An Overview of Recent Developmentscited by 37
  7. 2019Bridging the Gap Between Monaural Speech Enhancement and Recognition with Distortion-Independent Acoustic Modelingcited by 36
  8. 2019Espresso: A Fast End-to-end Neural Speech Recognition Toolkitcited by 34
  9. 2021Wav2vec-C: A Self-supervised Model for Speech Representation Learningcited by 29
  10. 2021Interactive Feature Fusion for End-to-End Noise-Robust Speech Recognitioncited by 26
  11. 2022SkiM: Skipping Memory LSTM for Low-Latency Real-Time Continuous Speech Separationcited by 20
  12. 2020Gated Recurrent Fusion with Joint Training Framework for Robust End-to-End Speech Recognitioncited by 17
  13. 2022Joint Training of Speech Enhancement and Self-supervised Model for Noise-robust ASRcited by 15

Built on by

Cited by year

Citing papers per year: 2024: 1; 2025: 1.

2 papers.

  1. 2025Improving Speech Enhancement with Multi-Metric Supervision from Learned Quality Assessmentcited by 7
  2. 2024Towards Decoupling Frontend Enhancement and Backend Recognition in Monaural Robust ASRcited by 5

Similar, not cited

Close in wording, no citation either way

  1. 2024MS-HuBERT: Mitigating Pre-training and Inference Mismatch in Masked Language Modelling methods for learning Speech Representations
  2. 2022An Adapter based Multi-label Pre-training for Speech Separation and Enhancement
  3. 2023Wav2code: Restore Clean Speech Representations via Codebook Lookup for Noise-Robust ASR
  4. 2022HuBERT-EE: Early Exiting HuBERT for Efficient Speech Recognition
  5. 2023Downstream Task Agnostic Speech Enhancement with Self-Supervised Representation Loss
  6. 2021An Investigation of End-to-End Models for Robust Speech Recognition
  7. 2022Boosting Self-Supervised Embeddings for Speech Enhancement
  8. 2026Masked Autoencoders as Universal Speech Enhancer