mlsanityData to 7 Oct 2026

Property Neurons in Self-Supervised Speech Transformers

arXiv 2409.05910 ↗First posted 7 Sep 2024eess.AS cs.CL cs.LGcited by 5

Not on the map. Why?

Abstract (arXiv)

There have been many studies on analyzing self-supervised speech Transformers, in particular, with layer-wise analysis. It is, however, desirable to have an approach that can pinpoint exactly a subset of neurons that is responsible for a particular property of speech, being amenable to model pruning and model editing. In this work, we identify a set of property neurons in the feedforward layers of Transformers to study how speech-related properties, such as phones, gender, and pitch, are stored. When removing neurons of a particular property (a simple form of model editing), the respective downstream performance significantly degrades, showing the importance of the property neurons. We apply this approach to pruning the feedforward layers in Transformers, where most of the model parameters are. We show that protecting property neurons during pruning is significantly more effective than norm-based pruning. The code for identifying property neurons is available at https://github.com/nervjack2/PropertyNeurons.

Key facts. Cited by 5 papers in the map (data through 7 Oct 2026). Builds on 24 of its 32 references that are in the map. First posted on arXiv 7 Sep 2024.

Builds on

24 of 32 references.

References · 24

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,853
  3. 2016Gaussian Error Linear Units (GELUs)cited by 3,431
  4. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,301
  5. 2021WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processingcited by 1,609
  6. 2020Transformer Feed-Forward Layers Are Key-Value Memoriescited by 1,307
  7. 2021SUPERB: Speech processing Universal PERformance Benchmarkcited by 652
  8. 2021Knowledge Neurons in Pretrained Transformerscited by 590
  9. 2021Layer-wise Analysis of a Self-supervised Speech Representation Modelcited by 293
  10. 2022Comparative layer-wise analysis of self-supervised speech modelscited by 155
  11. 2017Learning to Generate Reviews and Discovering Sentimentcited by 147
  12. 2018What Is One Grain of Sand in the Desert? Analyzing Individual Neurons in Deep NLP Modelscited by 124
12 more
  1. 2024Language-Specific Neurons: The Key to Multilingual Capabilities in Large Language Modelscited by 124
  2. 2018Identifying and Controlling Important Neurons in Neural Machine Translationcited by 110
  3. 2023Neurons in Large Language Models: Dead, N-gram, Positionalcited by 88
  4. 2022SUPERB-SG: Enhanced Speech processing Universal PERformance Benchmark for Semantic and Generative Capabilitiescited by 76
  5. 2022Finding Skill Neurons in Pre-trained Transformer-based Language Modelscited by 65
  6. 2023What Do Self-Supervised Speech Models Know About Words?cited by 62
  7. 2022On the Utility of Self-supervised Models for Prosody-related Taskscited by 42
  8. 2022SUPERB @ SLT 2022: Challenge on Generalization and Efficiency of Self-Supervised Speech Representation Learningcited by 26
  9. 2024Learnable Privacy Neurons Localization in Language Modelscited by 26
  10. 2022MelHuBERT: A simplified HuBERT on Mel spectrogramscited by 15
  11. 2024What Do Self-Supervised Speech and Speaker Models Learn? New Findings From a Cross Model Layer-Wise Analysiscited by 10
  12. 2022Is Smaller Always Faster? Tradeoffs in Compressing Self-Supervised Speech Transformerscited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 3; 2026: 2.

5 papers.

  1. 2025AudioLens: A Closer Look at Auditory Attribute Perception of Large Audio-Language Modelscited by 18
  2. 2026How Contrastive Decoding Enhances Large Audio Language Modelscited by 5
  3. 2026Causal Tracing of Audio-Text Fusion in Large Audio Language Modelscited by 4
  4. 2025An Exploration of Mamba for Speech Self-Supervised Modelscited by 2
  5. 2025Identifying Speaker Information in Feed-Forward Layers of Self-Supervised Speech Transformerscited by 1