mlsanityData to 7 Oct 2026

Audio ALBERT: A Lite BERT for Self-supervised Learning of Audio Representation

arXiv 2005.08575 ↗First posted 18 May 2020eess.AS cs.CL cs.SDcited by 58

Not on the map. Why?

Abstract (arXiv)

For self-supervised speech processing, it is crucial to use pretrained models as speech representation extractors. In recent works, increasing the size of the model has been utilized in acoustic model training in order to achieve better performance. In this paper, we propose Audio ALBERT, a lite version of the self-supervised speech representation model. We use the representations with two downstream tasks, speaker identification, and phoneme classification. We show that Audio ALBERT is capable of achieving competitive performance with those huge models in the downstream tasks while utilizing 91% fewer parameters. Moreover, we use some simple probing models to measure how much the information of the speaker and phoneme is encoded in latent representations. In probing experiments, we find that the latent representations encode richer information of both phoneme and speaker than that of the last layer.

Key facts. Cited by 58 papers in the map (data through 7 Oct 2026). First posted on arXiv 18 May 2020.

Builds on

No reference list yet.

Built on by

Cited by year

Citing papers per year: 2021: 15; 2022: 20; 2023: 8; 2024: 12; 2025: 3.

58 papers.

Sort:

By area:

  1. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,301
  2. 2021Generative Spoken Language Modeling from Raw Audiocited by 336
  3. 2022Self-Supervised Speech Representation Learning: A Reviewcited by 258
  4. 2021Pano-AVQA: Grounded Audio-Visual Question Answering on 360^circ Videoscited by 86
  5. 2021SLUE: New Benchmark Tasks for Spoken Language Understanding Evaluation on Natural Speechcited by 70
  6. 2022Audio Self-supervised Learning: A Surveycited by 65
  7. 2021What all do audio transformer models hear? Probing Acoustic Representations for Language Delivery and its Structurecited by 52
  8. 2021CDPAM: Contrastive learning for perceptual audio similaritycited by 42
  9. 2024A Large-Scale Evaluation of Speech Foundation Modelscited by 38
  10. 2021Textless Speech Emotion Conversion using Discrete and Decomposed Representationscited by 29
  11. 2022Beyond Just Vision: A Review on Self-Supervised Representation Learning on Multimodal and Temporal Datacited by 27
  12. 2022SUPERB @ SLT 2022: Challenge on Generalization and Efficiency of Self-Supervised Speech Representation Learningcited by 26
38 more
  1. 2024Wings: Learning Multimodal LLMs without Text-only Forgettingcited by 23
  2. 2024Bypassing the Exponential Dependency: Looped Transformers Efficiently Learn In-context by Multi-step Gradient Descentcited by 21
  3. 2021A Survey on Deep Reinforcement Learning for Audio-Based Applicationscited by 17
  4. 2021Improving the Adversarial Robustness for Speaker Verification by Self-Supervised Learningcited by 16
  5. 2023Masked Modeling for Self-supervised Representation Learning on Vision and Beyondcited by 13
  6. 2021One4all User Representation for Recommender Systems in E-commercecited by 12
  7. 2022A Pre-trained Audio-Visual Transformer for Emotion Recognitioncited by 12
  8. 2021What do End-to-End Speech Models Learn about Speaker, Language and Channel Information? A Layer-wise and Neuron-level Analysiscited by 9
  9. 2021CTAL: Pre-training Cross-modal Transformer for Audio-and-Language Representationscited by 9
  10. 2022A Brief Overview of Unsupervised Neural Speech Representation Learningcited by 9
  11. 2022Learning Representations for New Sound Classes With Continual Self-Supervised Learningcited by 8
  12. 2022StorSeismic: A new paradigm in deep learning for seismic processingcited by 7
  13. 2022Silence is Sweeter Than Speech: Self-Supervised Model Using Silence to Store Speaker Informationcited by 7
  14. 2022Losses Can Be Blessings: Routing Self-Supervised Speech Representations Towards Efficient Multilingual and Multitask Speech Processingcited by 7
  15. 2022Distribution Aware Metrics for Conditional Natural Language Generationcited by 6
  16. 2022Match to Win: Analysing Sequences Lengths for Efficient Self-supervised Learning in Speech and Audiocited by 6
  17. 2023Recycle-and-Distill: Universal Compression Strategy for Transformer-based Speech SSL Models with Attention Map Reusing and Masking Distillationcited by 6
  18. 2022Towards Proper Contrastive Self-supervised Learning Strategies For Music Audio Representationcited by 5
  19. 2024Efficient Training of Self-Supervised Speech Foundation Models on a Compute Budgetcited by 5
  20. 2022Self-Supervised Audio-and-Text Pre-training with Extremely Low-Resource Parallel Datacited by 4
  21. 2024Head-wise Shareable Attention for Large Language Modelscited by 4
  22. 2022Improving Label-Deficient Keyword Spotting Through Self-Supervised Pretrainingcited by 3
  23. 2023Looking Similar, Sounding Different: Leveraging Counterfactual Cross-Modal Pairs for Audiovisual Representation Learningcited by 3
  24. 2023A Comparative Study of Pre-trained Speech and Audio Embeddings for Speech Emotion Recognitioncited by 3
  25. 2022Improving Children's Speech Recognition by Fine-tuning Self-supervised Adult Speech Representationscited by 2
  26. 2022Survey on Self-Supervised Multimodal Representation Learning and Foundation Modelscited by 2
  27. 2023A Quantitative Approach to Understand Self-Supervised Models as Cross-lingual Feature Extractorscited by 2
  28. 2024On the Transferability of Large-Scale Self-Supervision to Few-Shot Audio Classificationcited by 2
  29. 2024Audio Explanation Synthesis with Generative Foundation Modelscited by 2
  30. 2025HierPromptLM: A Pure PLM-based Framework for Representation Learning on Heterogeneous Text-rich Networkscited by 2
  31. 2021Using Pause Information for More Accurate Entity Recognitioncited by 1
  32. 2022Spatio-Temporal Analysis of Transformer based Architecture for Attention Estimation from EEGcited by 1
  33. 2023A Comparison of Speech Data Augmentation Methods Using S3PRL Toolkitcited by 1
  34. 2023Acoustic and linguistic representations for speech continuous emotion recognition in call center conversationscited by 1
  35. 2024Deep Learning for Assessment of Oral Reading Fluencycited by 1
  36. 2024Speech Separation with Pretrained Frontend to Minimize Domain Mismatchcited by 1
  37. 2021Label-Synchronous Speech-to-Text Alignment for ASR Using Forward and Backward Transformerscited by 0
  38. 2021Layer Reduction: Accelerating Conformer-Based Self-Supervised Model via Layer Consistencycited by 0

50 most cited of 58 All 58 (JSON)