mlsanityData to 7 Oct 2026

Efficient Speech Representation Learning with Low-Bit Quantization

arXiv 2301.00652 ↗First posted 14 Dec 2022eess.AS cs.CLcited by 4

Not on the map. Why?

Abstract (arXiv)

With the development of hardware for machine learning, newer models often come at the cost of both increased sizes and computational complexity. In effort to improve the efficiency for these models, we apply and investigate recent quantization techniques on speech representation learning models. The quantization techniques were evaluated on the SUPERB benchmark. On the ASR task, with aggressive quantization to 1 bit, we achieved 86.32% storage reduction (184.42 -> 25.23), 88% estimated runtime reduction (1.00 -> 0.12) with increased word error rate (7.06 -> 15.96). In comparison with DistillHuBERT which also aims for model compression, the 2-bit configuration yielded slightly smaller storage (35.84 vs. 46.98), better word error rate (12.68 vs. 13.37) and more efficient estimated runtime (0.15 vs. 0.73).

Key facts. Cited by 4 papers in the map (data through 7 Oct 2026). Builds on 20 of its 23 references that are in the map. First posted on arXiv 14 Dec 2022.

Builds on

20 of 23 references.

References · 20

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  5. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  6. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,853
  7. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,301
  8. 2013Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computationcited by 2,031
  9. 2017Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inferencecited by 1,548
  10. 2019fairseq: A Fast, Extensible Toolkit for Sequence Modelingcited by 1,486
  11. 2016XNOR-Net: ImageNet Classification Using Binary Convolutional Neural Networkscited by 829
  12. 2021SUPERB: Speech processing Universal PERformance Benchmarkcited by 652
8 more
  1. 2021Pruning and Quantization for Deep Neural Network Acceleration: A Surveycited by 259
  2. 2020Training with Quantization Noise for Extreme Model Compressioncited by 145
  3. 2021DistilHuBERT: Speech Representation Learning by Layer-wise Distillation of Hidden-unit BERTcited by 129
  4. 2021TorchAudio: Building Blocks for Audio and Speech Processingcited by 83
  5. 2021Performance-Efficiency Trade-offs in Unsupervised Pre-training for Speech Recognitioncited by 68
  6. 2022BiT: Robustly Binarized Multi-distilled Transformercited by 51
  7. 2019QKD: Quantization-aware Knowledge Distillationcited by 42
  8. 2019QuaRL: Quantization for Fast and Environmentally Sustainable Reinforcement Learningcited by 19

Built on by

Cited by year

Citing papers per year: 2023: 3; 2024: 1.

4 papers.

  1. 2024OWSM-CTC: An Open Encoder-Only Speech Foundation Model for Speech Recognition, Translation, and Language Identificationcited by 34
  2. 2023Fine-tuning Strategies for Faster Inference using Speech Self-Supervised Models: A Comparative Studycited by 15
  3. 2023CoLLD: Contrastive Layer-to-layer Distillation for Compressing Multilingual Pre-trained Speech Encoderscited by 7
  4. 2023Attention or Convolution: Transformer Encoders in Audio Language Models for Inference Efficiencycited by 0