mlsanityData to 14 Oct 2026

One-pass Multiple Conformer and Foundation Speech Systems Compression and Quantization Using An All-in-one Neural Model

arXiv 2406.10160 ↗First posted 14 Jun 2024cs.SD cs.AI eess.AScited by 4

Not on the map. Why?

Abstract (arXiv)

We propose a novel one-pass multiple ASR systems joint compression and quantization approach using an all-in-one neural model. A single compression cycle allows multiple nested systems with varying Encoder depths, widths, and quantization precision settings to be simultaneously constructed without the need to train and store individual target systems separately. Experiments consistently demonstrate the multiple ASR systems compressed in a single all-in-one model produced a word error rate (WER) comparable to, or lower by up to 1.01% absolute (6.98% relative) than individually trained systems of equal complexity. A 3.4x overall system compression and training time speed-up was achieved. Maximum model size compression ratios of 12.8x and 3.93x were obtained over the baseline Switchboard-300hr Conformer and LibriSpeech-100hr fine-tuned wav2vec2.0 models, respectively, incurring no statistically significant WER increase.

Key facts. Cited by 4 papers in the map (data through 14 Oct 2026). Builds on 38 of its 47 references that are in the map. First posted on arXiv 14 Jun 2024.

Builds on

38 of 47 references.

References · 38

  1. 2017Attention Is All You Needcited by 60,361
  2. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 4,057
  3. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,905
  4. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,342
  5. 2020Conformer: Convolution-augmented Transformer for Speech Recognitioncited by 1,863
  6. 2021WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processingcited by 1,657
  7. 2019Once-for-All: Train One Network and Specialize it for Efficient Deploymentcited by 787
  8. 2022data2vec: A General Framework for Self-supervised Learning in Speech, Vision and Languagecited by 727
  9. 2018ESPnet: End-to-End Speech Processing Toolkitcited by 558
  10. 2021A White Paper on Neural Network Quantizationcited by 472
  11. 2020DynaBERT: Dynamic BERT with Adaptive Width and Depthcited by 251
  12. 2019Universally Slimmable Networks and Improved Training Techniquescited by 221
26 more
  1. 2020BigNAS: Scaling Up Neural Architecture Search with Big Single-Stage Modelscited by 188
  2. 2021DistilHuBERT: Speech Representation Learning by Layer-wise Distillation of Hidden-unit BERTcited by 144
  3. 2022Branchformer: Parallel MLP-Attention Architectures to Capture Local and Global Context for Speech Recognition and Understandingcited by 91
  4. 2021PARP: Prune, Adjust and Re-Prune for Self-Supervised Speech Recognitioncited by 53
  5. 2022LightHuBERT: Lightweight and Configurable Speech Representation Learning with Once-for-All Hidden-Unit BERTcited by 50
  6. 2023DPHuBERT: Joint Distillation and Pruning of Self-Supervised Speech Modelscited by 36
  7. 2022Deep versus Wide: An Analysis of Student Architectures for Task-Agnostic Knowledge Distillation of Self-Supervised Speech Modelscited by 31
  8. 2023Structured Pruning of Self-Supervised Pre-trained Models for Speech Recognition and Understandingcited by 31
  9. 2022Generalizing Few-Shot NAS with Gradient Matchingcited by 25
  10. 2022FitHuBERT: Going Thinner and Deeper for Knowledge Distillation of Speech Self-Supervised Learningcited by 25
  11. 20224-bit Conformer with Native Quantization Aware Training for Speech Recognitioncited by 24
  12. 2020Dynamic Sparsity Neural Networks for Automatic Speech Recognitioncited by 23
  13. 2021Integer-only Zero-shot Quantization for Efficient Speech Recognitioncited by 19
  14. 2021Efficient conformer-based speech recognition with linear attentioncited by 13
  15. 2021Layer Pruning on Demand with Intermediate CTCcited by 13
  16. 2022PADA: Pruning Assisted Domain Adaptation for Self-Supervised Speech Representationscited by 11
  17. 2023Accurate and Structured Pruning for Efficient Automatic Speech Recognitioncited by 11
  18. 2021Mixed Precision of Quantization of Transformer Language Models for Speech Recognitioncited by 10
  19. 20232-bit Conformer quantization for automatic speech recognitioncited by 10
  20. 2022Multi-stage Progressive Compression of Conformer Transducer for On-device Speech Recognitioncited by 9
  21. 2023CoLLD: Contrastive Layer-to-layer Distillation for Compressing Multilingual Pre-trained Speech Encoderscited by 9
  22. 2023USM-Lite: Quantization and Sparsity Aware Fine-tuning for Speech Recognition with Universal Speech Modelscited by 9
  23. 2023A Model for Every User and Budget: Label-Free and Personalized Mixed-Precision Quantizationcited by 8
  24. 2023DistillW2V2: A Small and Streaming Wav2vec 2.0 Based ASR Modelcited by 7
  25. 2023Distilling HuBERT with LSTMs via Decoupled Knowledge Distillationcited by 4
  26. 2022Once-for-All Sequence Compression for Self-Supervised Speech Modelscited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 2; 2026: 2.

4 papers.

  1. 2025Effective and Efficient Mixed Precision Quantization of Speech Foundation Modelscited by 5
  2. 2025Effective and Efficient One-pass Compression of Speech Foundation Models Using Sparsity-aware Self-pinching Gatescited by 1
  3. 2026Diagnostic-Driven Layer-Wise Compensation for Post-Training Quantization of Encoder-Decoder ASR Modelscited by 0
  4. 2026Towards Data-free and Training-free Compression for Speech Foundation Models Using Parameter Clusteringcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Unfolding A Few Structures for The Many: Memory-Efficient Compression of Conformer and Speech Foundation Models
  2. 2022Sub-8-bit quantization for on-device speech recognition: a regularization-free approach
  3. 2023On the compression of shallow non-causal ASR models using knowledge distillation and tied-and-reduced decoder for low-latency on-device speech recognition
  4. 2025HH-Codec: High Compression High-fidelity Discrete Neural Codec for Spoken Language Modeling
  5. 2022Efficient Speech Representation Learning with Low-Bit Quantization
  6. 2021Investigation of Practical Aspects of Single Channel Speech Separation for ASR
  7. 2025Edge-ASR: Towards Low-Bit Quantization of Automatic Speech Recognition Models
  8. 2024Self-Supervised Learning for Multi-Channel Neural Transducer