mlsanityData to 14 Oct 2026

Joint Unsupervised and Supervised Training for Automatic Speech Recognition via Bilevel Optimization

arXiv 2401.06980 ↗First posted 13 Jan 2024cs.CL cs.LG stat.MLcited by 5

Not on the map. Why?

Abstract (arXiv)

In this paper, we present a novel bilevel optimization-based training approach to training acoustic models for automatic speech recognition (ASR) tasks that we term {bi-level joint unsupervised and supervised training (BL-JUST)}. {BL-JUST employs a lower and upper level optimization with an unsupervised loss and a supervised loss respectively, leveraging recent advances in penalty-based bilevel optimization to solve this challenging ASR problem with affordable complexity and rigorous convergence guarantees.} To evaluate BL-JUST, extensive experiments on the LibriSpeech and TED-LIUM v2 datasets have been conducted. BL-JUST achieves superior performance over the commonly used pre-training followed by fine-tuning strategy.

Key facts. Cited by 5 papers in the map (data through 14 Oct 2026). Builds on 20 of its 33 references that are in the map. First posted on arXiv 13 Jan 2024.

Builds on

20 of 33 references.

References · 20

  1. 2018Representation Learning with Contrastive Predictive Codingcited by 7,777
  2. 2017Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networkscited by 4,897
  3. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,905
  4. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,342
  5. 2020Conformer: Convolution-augmented Transformer for Speech Recognitioncited by 1,863
  6. 2019SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognitioncited by 1,380
  7. 2016Optimization Methods for Large-Scale Machine Learningcited by 1,150
  8. 2018Bilevel Programming for Hyperparameter Optimization and Meta-Learningcited by 405
  9. 2019vq-wav2vec: Self-Supervised Learning of Discrete Speech Representationscited by 340
  10. 2016Hyperparameter optimization with approximate gradientcited by 233
  11. 2020Loss landscapes and optimization in over-parameterized non-linear systems and neural networkscited by 233
  12. 2018Characterizing and Avoiding Negative Transfercited by 229
8 more
  1. 2022Self-supervised Learning with Random-projection Quantizer for Speech Recognitioncited by 162
  2. 2021Investigating Bi-Level Optimization for Learning and Vision from a Unified Perspective: A Survey and Beyondcited by 153
  3. 2023On Penalty-based Bilevel Gradient Descent Methodcited by 87
  4. 2021A Single-Timescale Method for Stochastic Bilevel Optimizationcited by 69
  5. 2017A First Order Method for Solving Convex Bi-Level Optimization Problemscited by 67
  6. 2021Joint Unsupervised and Supervised Training for Multilingual ASRcited by 39
  7. 2021Bilevel methods for image reconstructioncited by 20
  8. 2022Learning with Limited Samples -- Meta-Learning and Applications to Communication Systemscited by 13

Built on by

Cited by year

Citing papers per year: 2024: 2; 2025: 3.

5 papers.

  1. 2024Whistle: Data-Efficient Multilingual and Crosslingual Speech Recognition via Weakly Phonetic Supervisioncited by 17
  2. 2025A Fully First-Order Layer for Differentiable Optimizationcited by 3
  3. 2024Bilevel Joint Unsupervised and Supervised Training for Automatic Speech Recognitioncited by 1
  4. 2025Objective Soups: Multilingual Multi-Task Modeling for Speech Processingcited by 0
  5. 2025BiRQ: Bi-Level Self-Labeling Random Quantization for Self-Supervised Speech Recognitioncited by 0