# Interpretable Embeddings of Speech Enhance and Explain Brain Encoding Performance of Audio Models

arXiv: https://arxiv.org/abs/2507.16080 · HTML: https://mlsanity.com/paper/2507.16080/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 21 Jul 2025.

## Built on by (most cited first, 2 of 2)

- [Mechanistic Interpretability of Brain-to-Speech Models Across Speech Modes](https://mlsanity.com/paper/2602.01247/) (2026)
- [Mechanistic Interpretability Reveals Shared Causal Subspaces in Brain-to-Speech Decoders](https://mlsanity.com/paper/2609.31992/) (2026)

## Abstract (arXiv)

Speech foundation models (SFMs) are increasingly hailed as powerful computational models of human speech perception. However, since their representations are inherently black-box, it remains unclear what drives their alignment with brain responses. To remedy this, we built linear encoding models from six interpretable feature families: mel-spectrogram, Gabor filter bank features, speech presence, phonetic, syntactic, and semantic features, and contextualized embeddings from three state-of-the-art SFMs (Whisper, HuBERT, WavLM), quantifying electrocorticography (ECoG) response variance shared between feature classes. Variance-partitioning analyses revealed several key insights: First, the SFMs' alignment with the brain can be mostly explained by their ability to learn and encode simple interpretable speech features. Second, SFMs exhibit a systematic trade-off between encoding of brain-relevant low-level and high-level features across layers. Finally, our results show that SFMs learn brain-relevant semantics which cannot be explained by lower-level speech features, with this capacity increasing with model size and context length. Together, our findings suggest a principled approach to build more interpretable, accurate, and efficient encoding models of the brain by augmenting SFM embeddings with interpretable features.