Understanding Self-Attention of Self-Supervised Audio Transformers
Not on the map. Why?
Abstract (arXiv)
Self-supervised Audio Transformers (SAT) enable great success in many downstream speech applications like ASR, but how they work has not been widely explored yet. In this work, we present multiple strategies for the analysis of attention mechanisms in SAT. We categorize attentions into explainable categories, where we discover each category possesses its own unique functionality. We provide a visualization tool for understanding multi-head self-attention, importance ranking strategies for identifying critical attention, and attention refinement techniques to improve model performance.
Key facts. Cited by 15 papers in the map (data through 7 Oct 2026). First posted on arXiv 5 Jun 2020.
Builds on
No reference list yet.
Built on by
Citing papers per year: 2021: 1; 2022: 7; 2023: 2; 2024: 3; 2025: 2.
15 papers.
Sort:
- 2022Self-Supervised Speech Representation Learning: A Reviewcited by 258
- 2022Audio Self-supervised Learning: A Surveycited by 65
- 2022Deep versus Wide: An Analysis of Student Architectures for Task-Agnostic Knowledge Distillation of Self-Supervised Speech Modelscited by 28
- 2021Don't speak too fast: The impact of data bias on self-supervised speech modelscited by 20
- 2024What Do Self-Supervised Speech and Speaker Models Learn? New Findings From a Cross Model Layer-Wise Analysiscited by 10
- 2022Topological Data Analysis for Speech Processingcited by 9
- 2024Efficient Training of Self-Supervised Speech Foundation Models on a Compute Budgetcited by 5
- 2025Whisper Has an Internal Word Alignercited by 5
- 2025Layer-wise Minimal Pair Probing Reveals Contextual Grammatical-Conceptual Hierarchy in Speech Representationscited by 5
- 2022Similarity and Content-based Phonetic Self Attention for Speech Recognitioncited by 3
- 2023Exploring Attention Map Reuse for Efficient Transformer Neural Networkscited by 3
- 2024Speaker-Independent Dysarthria Severity Classification using Self-Supervised Transformers and Multi-Task Learningcited by 2
3 more
- 2022DEFORMER: Coupling Deformed Localized Patterns with Global Context for Robust End-to-end Speech Recognitioncited by 0
- 2022A Comparison of Transformer, Convolutional, and Recurrent Neural Networks on Phoneme Recognitioncited by 0
- 2023Knowledge Distillation from Non-streaming to Streaming ASR Encoder using Auxiliary Non-streaming Layercited by 0
- 2025Layer-wise Minimal Pair Probing Reveals Contextual Grammatical-Conceptual Hierarchy in Speech Representations19 Sep 2025
- 2025Whisper Has an Internal Word Aligner12 Sep 2025
- 2024Efficient Training of Self-Supervised Speech Foundation Models on a Compute Budget9 Sep 2024
- 2024Speaker-Independent Dysarthria Severity Classification using Self-Supervised Transformers and Multi-Task Learning29 Feb 2024
- 2024What Do Self-Supervised Speech and Speaker Models Learn? New Findings From a Cross Model Layer-Wise Analysis31 Jan 2024
- 2023Knowledge Distillation from Non-streaming to Streaming ASR Encoder using Auxiliary Non-streaming Layer31 Aug 2023
- 2023Exploring Attention Map Reuse for Efficient Transformer Neural Networks29 Jan 2023
- 2022Topological Data Analysis for Speech Processing30 Nov 2022
- 2022A Comparison of Transformer, Convolutional, and Recurrent Neural Networks on Phoneme Recognition1 Oct 2022
- 2022Deep versus Wide: An Analysis of Student Architectures for Task-Agnostic Knowledge Distillation of Self-Supervised Speech Models14 Jul 2022
- 2022DEFORMER: Coupling Deformed Localized Patterns with Global Context for Robust End-to-end Speech Recognition4 Jul 2022
- 2022Self-Supervised Speech Representation Learning: A Review21 May 2022
3 more
- 2022Self-Supervised Speech Representation Learning: A Reviewcited by 258
- 2022Audio Self-supervised Learning: A Surveycited by 65
- 2022Deep versus Wide: An Analysis of Student Architectures for Task-Agnostic Knowledge Distillation of Self-Supervised Speech Modelscited by 28
- 2021Don't speak too fast: The impact of data bias on self-supervised speech modelscited by 20
- 2024What Do Self-Supervised Speech and Speaker Models Learn? New Findings From a Cross Model Layer-Wise Analysiscited by 10
- 2022Topological Data Analysis for Speech Processingcited by 9
- 2024Efficient Training of Self-Supervised Speech Foundation Models on a Compute Budgetcited by 5
- 2025Whisper Has an Internal Word Alignercited by 5
- 2025Layer-wise Minimal Pair Probing Reveals Contextual Grammatical-Conceptual Hierarchy in Speech Representationscited by 5
- 2022Similarity and Content-based Phonetic Self Attention for Speech Recognitioncited by 3
- 2023Exploring Attention Map Reuse for Efficient Transformer Neural Networkscited by 3
- 2024Speaker-Independent Dysarthria Severity Classification using Self-Supervised Transformers and Multi-Task Learningcited by 2
12 most cited of 15 All 15 (JSON)
- 2025Layer-wise Minimal Pair Probing Reveals Contextual Grammatical-Conceptual Hierarchy in Speech Representations19 Sep 2025
- 2025Whisper Has an Internal Word Aligner12 Sep 2025
- 2024Efficient Training of Self-Supervised Speech Foundation Models on a Compute Budget9 Sep 2024
- 2024Speaker-Independent Dysarthria Severity Classification using Self-Supervised Transformers and Multi-Task Learning29 Feb 2024
- 2024What Do Self-Supervised Speech and Speaker Models Learn? New Findings From a Cross Model Layer-Wise Analysis31 Jan 2024
- 2023Knowledge Distillation from Non-streaming to Streaming ASR Encoder using Auxiliary Non-streaming Layer31 Aug 2023
- 2023Exploring Attention Map Reuse for Efficient Transformer Neural Networks29 Jan 2023
- 2022Topological Data Analysis for Speech Processing30 Nov 2022
- 2022A Comparison of Transformer, Convolutional, and Recurrent Neural Networks on Phoneme Recognition1 Oct 2022
- 2022Deep versus Wide: An Analysis of Student Architectures for Task-Agnostic Knowledge Distillation of Self-Supervised Speech Models14 Jul 2022
- 2022DEFORMER: Coupling Deformed Localized Patterns with Global Context for Robust End-to-end Speech Recognition4 Jul 2022
- 2022Self-Supervised Speech Representation Learning: A Review21 May 2022
12 newest of 15 All 15 (JSON)