mlsanityData to 7 Oct 2026

CoLLD: Contrastive Layer-to-layer Distillation for Compressing Multilingual Pre-trained Speech Encoders

arXiv 2309.07707 ↗First posted 14 Sep 2023cs.CL cs.SD eess.AScited by 7

Not on the map. Why?

Abstract (arXiv)

Large-scale self-supervised pre-trained speech encoders outperform conventional approaches in speech recognition and translation tasks. Due to the high cost of developing these large models, building new encoders for new tasks and deploying them to on-device applications are infeasible. Prior studies propose model compression methods to address this issue, but those works focus on smaller models and less realistic tasks. Thus, we propose Contrastive Layer-to-layer Distillation (CoLLD), a novel knowledge distillation method to compress pre-trained speech encoders by leveraging masked prediction and contrastive learning to train student models to copy the behavior of a large teacher model. CoLLD outperforms prior methods and closes the gap between small and large models on multilingual speech-to-text translation and recognition benchmarks.

Key facts. Cited by 7 papers in the map (data through 7 Oct 2026). Builds on 37 of its 38 references that are in the map. First posted on arXiv 14 Sep 2023.

Builds on

37 of 38 references.

References · 37

  1. 2017Attention Is All You Needcited by 59,932
  2. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  3. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  4. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,853
  5. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,301
  6. 2020Conformer: Convolution-augmented Transformer for Speech Recognitioncited by 1,838
  7. 2019fairseq: A Fast, Extensible Toolkit for Sequence Modelingcited by 1,486
  8. 2022No Language Left Behind: Scaling Human-Centered Machine Translationcited by 1,290
  9. 2022data2vec: A General Framework for Self-supervised Learning in Speech, Vision and Languagecited by 718
  10. 2021SUPERB: Speech processing Universal PERformance Benchmarkcited by 652
  11. 2021XLS-R: Self-supervised Cross-lingual Speech Representation Learning at Scalecited by 589
  12. 2018ESPnet: End-to-End Speech Processing Toolkitcited by 554
25 more
  1. 2020Unsupervised Cross-lingual Representation Learning for Speech Recognitioncited by 496
  2. 2022FLEURS: Few-shot Learning Evaluation of Universal Representations of Speechcited by 458
  3. 2023Scaling Speech Technology to 1,000+ Languagescited by 401
  4. 2022Self-Supervised Speech Representation Learning: A Reviewcited by 258
  5. 2023Google USM: Scaling Automatic Speech Recognition Beyond 100 Languagescited by 228
  6. 2021DistilHuBERT: Speech Representation Learning by Layer-wise Distillation of Hidden-unit BERTcited by 129
  7. 2023SeamlessM4T: Massively Multilingual & Multimodal Machine Translationcited by 124
  8. 2022SUPERB-SG: Enhanced Speech processing Universal PERformance Benchmark for Semantic and Generative Capabilitiescited by 76
  9. 2023ML-SUPERB: Multilingual Speech Universal PERformance Benchmarkcited by 70
  10. 2021PARP: Prune, Adjust and Re-Prune for Self-Supervised Speech Recognitioncited by 50
  11. 2022LightHuBERT: Lightweight and Configurable Speech Representation Learning with Once-for-All Hidden-Unit BERTcited by 45
  12. 2023DPHuBERT: Joint Distillation and Pruning of Self-Supervised Speech Modelscited by 32
  13. 2023Structured Pruning of Self-Supervised Pre-trained Models for Speech Recognition and Understandingcited by 29
  14. 2022Deep versus Wide: An Analysis of Student Architectures for Task-Agnostic Knowledge Distillation of Self-Supervised Speech Modelscited by 28
  15. 2022SUPERB @ SLT 2022: Challenge on Generalization and Efficiency of Self-Supervised Speech Representation Learningcited by 26
  16. 2022FitHuBERT: Going Thinner and Deeper for Knowledge Distillation of Speech Self-Supervised Learningcited by 24
  17. 2023Self-supervised Fine-tuning for Improved Content Representations by Speaker-invariant Clusteringcited by 21
  18. 2022M-Adapter: Modality Adaptation for End-to-End Speech-to-Text Translationcited by 15
  19. 2023Task-Agnostic Structured Pruning of Speech Representation Modelscited by 13
  20. 2023Ensemble knowledge distillation of self-supervised speech modelscited by 10
  21. 2023I3D: Transformer architectures with input-dependent dynamic depth for speech recognitioncited by 10
  22. 2023Accurate and Structured Pruning for Efficient Automatic Speech Recognitioncited by 10
  23. 2023Recycle-and-Distill: Universal Compression Strategy for Transformer-based Speech SSL Models with Attention Map Reusing and Masking Distillationcited by 6
  24. 2022Efficient Speech Representation Learning with Low-Bit Quantizationcited by 4
  25. 2023DistilXLSR: A Light Weight Cross-Lingual Speech Representation Modelcited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 2; 2025: 3; 2026: 2.

7 papers.

  1. 2024Applications of Knowledge Distillation in Remote Sensing: A Surveycited by 8
  2. 2025USAD: Universal Speech and Audio Representation via Distillationcited by 7
  3. 2025Effective and Efficient Mixed Precision Quantization of Speech Foundation Modelscited by 5
  4. 2024One-pass Multiple Conformer and Foundation Speech Systems Compression and Quantization Using An All-in-one Neural Modelcited by 4
  5. 2025Speech-FT: Merging Pre-trained And Fine-Tuned Speech Representation Models For Cross-Task Generalizationcited by 2
  6. 2026Alethia: A Foundational Encoder for Voice Deepfakescited by 1
  7. 2026USAD 2.0: Scaling Representation Distillation for Universal Audio Understandingcited by 0