mlsanityData to 14 Oct 2026

USAD: Universal Speech and Audio Representation via Distillation

arXiv 2506.18843 ↗First posted 23 Jun 2025cs.SD cs.CL eess.AScited by 7

Not on the map. Why?

Abstract (arXiv)

Self-supervised learning (SSL) has revolutionized audio representations, yet models often remain domain-specific, focusing on either speech or non-speech tasks. In this work, we present Universal Speech and Audio Distillation (USAD), a unified approach to audio representation learning that integrates diverse audio types - speech, sound, and music - into a single model. USAD employs efficient layer-to-layer distillation from domain-specific SSL models to train a student on a comprehensive audio dataset. USAD offers competitive performance across various benchmarks and datasets, including frame and instance-level speech processing tasks, audio tagging, and sound classification, achieving near state-of-the-art results with a single encoder on SUPERB and HEAR benchmarks.

Key facts. Cited by 7 papers in the map (data through 14 Oct 2026). Builds on 53 of its 61 references that are in the map. First posted on arXiv 23 Jun 2025.

Builds on

53 of 61 references.

References · 53

  1. 2017Attention Is All You Needcited by 60,361
  2. 2016Layer Normalizationcited by 5,669
  3. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 4,057
  4. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,905
  5. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,342
  6. 2021WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processingcited by 1,657
  7. 2019fairseq: A Fast, Extensible Toolkit for Sequence Modelingcited by 1,488
  8. 2019Common Voice: A Massively-Multilingual Speech Corpuscited by 1,384
  9. 2018Self-Attention with Relative Position Representationscited by 1,120
  10. 2018Speech Commands: A Dataset for Limited-Vocabulary Speech Recognitioncited by 752
  11. 2022data2vec: A General Framework for Self-supervised Learning in Speech, Vision and Languagecited by 727
  12. 2022Large-scale Contrastive Language-Audio Pretraining with Feature Fusion and Keyword-to-Caption Augmentationcited by 719
38 more
  1. 2021SUPERB: Speech processing Universal PERformance Benchmarkcited by 666
  2. 2024Moshi: a speech-text foundation model for real-time dialoguecited by 568
  3. 2024Qwen2-Audio Technical Reportcited by 560
  4. 2023SALMONN: Towards Generic Hearing Abilities for Large Language Modelscited by 506
  5. 2020MLS: A Large-Scale Multilingual Dataset for Speech Researchcited by 476
  6. 2019Libri-Light: A Benchmark for ASR with Limited or No Supervisioncited by 467
  7. 2021VoxPopuli: A Large-Scale Multilingual Speech Corpus for Representation Learning, Semi-Supervised Learning and Interpretationcited by 463
  8. 2021GigaSpeech: An Evolving, Multi-domain ASR Corpus with 10,000 Hours of Transcribed Audiocited by 399
  9. 2022BEATs: Audio Pre-Training with Acoustic Tokenizerscited by 388
  10. 2021Layer-wise Analysis of a Self-supervised Speech Representation Modelcited by 298
  11. 2022Masked Autoencoders that Listencited by 273
  12. 2016SoundNet: Learning Sound Representations from Unlabeled Videocited by 247
  13. 2023Google USM: Scaling Automatic Speech Recognition Beyond 100 Languagescited by 231
  14. 2023Listen, Think, and Understandcited by 209
  15. 2023MERT: Acoustic Music Understanding Model with Large-Scale Self-supervised Trainingcited by 181
  16. 2021SSAST: Self-Supervised Audio Spectrogram Transformercited by 169
  17. 2024GAMA: A Large Audio-Language Model with Advanced Audio Understanding and Complex Reasoning Abilitiescited by 156
  18. 2021DistilHuBERT: Speech Representation Learning by Layer-wise Distillation of Hidden-unit BERTcited by 144
  19. 2020VoxLingua107: a Dataset for Spoken Language Recognitioncited by 122
  20. 2023SoundStorm: Efficient Parallel Audio Generationcited by 114
  21. 2022Efficient Self-supervised Learning with Contextualized Target Representations for Vision, Speech and Languagecited by 110
  22. 2021BYOL for Audio: Self-Supervised Learning for General-Purpose Audio Representationcited by 106
  23. 2023Joint Audio and Speech Understandingcited by 100
  24. 2022MAE-AST: Masked Autoencoding Audio Spectrogram Transformercited by 82
  25. 2022HEAR: Holistic Evaluation of Audio Representationscited by 78
  26. 2022SUPERB-SG: Enhanced Speech processing Universal PERformance Benchmark for Semantic and Generative Capabilitiescited by 77
  27. 2024EAT: Self-Supervised Pre-Training with Efficient Audio Transformercited by 53
  28. 2022Masked Spectrogram Modeling using Masked Autoencoders for Learning General-purpose Audio Representationcited by 52
  29. 2024A Large-Scale Evaluation of Speech Foundation Modelscited by 40
  30. 2023DinoSR: Self-Distillation and Online Clustering for Self-supervised Speech Representation Learningcited by 37
  31. 2023DPHuBERT: Joint Distillation and Pruning of Self-Supervised Speech Modelscited by 36
  32. 2022SUPERB @ SLT 2022: Challenge on Generalization and Efficiency of Self-Supervised Speech Representation Learningcited by 26
  33. 2023Self-supervised Audio Teacher-Student Transformer for Both Clip-level and Frame-level Taskscited by 25
  34. 2022Masked Modeling Duo: Learning Representations by Encouraging Both Networks to Model the Inputcited by 23
  35. 2025SSLAM: Enhancing Self-Supervised Models with Audio Mixtures for Polyphonic Soundscapescited by 16
  36. 2023Ensemble knowledge distillation of self-supervised speech modelscited by 12
  37. 2023CoLLD: Contrastive Layer-to-layer Distillation for Compressing Multilingual Pre-trained Speech Encoderscited by 9
  38. 2024DC-Spin: A Speaker-invariant Speech Tokenizer for Spoken Language Modelscited by 9

50 most cited of 53 All 53 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 2; 2026: 5.

7 papers.

  1. 2025Pushing the Frontier of Audiovisual Perception with Large-Scale Multimodal Correspondence Learningcited by 13
  2. 2025Curved Worlds, Clear Boundaries: Generalizing Speech Deepfake Detection using Hyperbolic and Spherical Geometry Spacescited by 1
  3. 2026Alethia: A Foundational Encoder for Voice Deepfakescited by 1
  4. 2026USAD 2.0: Scaling Representation Distillation for Universal Audio Understandingcited by 1
  5. 2026Stage-adaptive audio diffusion modelingcited by 0
  6. 2026ImmersiveTTS: Environment-Aware Text-to-Speech with Multimodal Diffusion Transformer and Domain-Specific Representation Alignmentcited by 0
  7. 2026GLaS-JEPA: Gaussian-Regularized Speech SSL without Engineered Prediction Targetscited by 0