# Self-Supervised Models of Speech Infer Universal Articulatory Kinematics

arXiv: https://arxiv.org/abs/2310.10788 · HTML: https://mlsanity.com/paper/2310.10788/

## Key facts

- Cited by 16 papers in the map (data through 14 Oct 2026).
- Builds on 14 of its 24 references that are in the map.
- First posted on arXiv 16 Oct 2023.

## Builds on (most cited first, 14 of 14)

- [wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations](https://mlsanity.com/paper/2006.11477/) (2020)
- [HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Units](https://mlsanity.com/paper/2106.07447/) (2021)
- [SUPERB: Speech processing Universal PERformance Benchmark](https://mlsanity.com/paper/2105.01051/) (2021)
- [XLS-R: Self-supervised Cross-lingual Speech Representation Learning at Scale](https://mlsanity.com/paper/2111.09296/) (2021)
- [Scaling Speech Technology to 1,000+ Languages](https://mlsanity.com/paper/2305.13516/) (2023)
- [Layer-wise Analysis of a Self-supervised Speech Representation Model](https://mlsanity.com/paper/2107.04734/) (2021)
- [Comparative layer-wise analysis of self-supervised speech models](https://mlsanity.com/paper/2211.03929/) (2022)
- [What all do audio transformer models hear? Probing Acoustic Representations for Language Delivery and its Structure](https://mlsanity.com/paper/2101.00387/) (2021)
- [Analysing Discrete Self Supervised Speech Representation for Spoken Language Modeling](https://mlsanity.com/paper/2301.00591/) (2023)
- [Evidence of Vocal Tract Articulation in Self-Supervised Learning of Speech](https://mlsanity.com/paper/2210.11723/) (2022)
- [Speaker-Independent Acoustic-to-Articulatory Speech Inversion](https://mlsanity.com/paper/2302.06774/) (2023)
- [An Information-Theoretic Analysis of Self-supervised Discrete Representations of Speech](https://mlsanity.com/paper/2306.02405/) (2023)
- [Deep Neural Convolutive Matrix Factorization for Articulatory Representation Decomposition](https://mlsanity.com/paper/2204.00465/) (2022)
- [Deep Speech Synthesis from Articulatory Representations](https://mlsanity.com/paper/2209.06337/) (2022)

## Built on by (most cited first, 16 of 16)

- [Sylber: Syllabic Embedding Representation of Speech from Raw Audio](https://mlsanity.com/paper/2410.07168/) (2024)
- [Coding Speech through Vocal Tract Kinematics](https://mlsanity.com/paper/2406.12998/) (2024)
- [YOLO-Stutter: End-to-end Region-Wise Speech Dysfluency Detection](https://mlsanity.com/paper/2408.15297/) (2024)
- [SSDM: Scalable Speech Dysfluency Modeling](https://mlsanity.com/paper/2408.16221/) (2024)
- [[b] = [d] - [t] + [p]: Self-supervised Speech Models Discover Phonological Vector Arithmetic](https://mlsanity.com/paper/2602.18899/) (2026)
- [Self-Supervised Speech Models Encode Phonetic Context via Position-dependent Orthogonal Subspaces](https://mlsanity.com/paper/2603.12642/) (2026)
- [RT-VC: Real-Time Zero-Shot Voice Conversion with Speech Articulatory Coding](https://mlsanity.com/paper/2506.10289/) (2025)
- [PRiSM: Benchmarking Phone Realization in Speech Models](https://mlsanity.com/paper/2601.14046/) (2026)
- [Fast, High-Quality and Parameter-Efficient Articulatory Synthesis using Differentiable DSP](https://mlsanity.com/paper/2409.02451/) (2024)
- [LinearVC: Linear transformations of self-supervised features through the lens of voice conversion](https://mlsanity.com/paper/2506.01510/) (2025)
- [emg2speech: Synthesizing speech from electromyography using self-supervised speech models](https://mlsanity.com/paper/2510.23969/) (2025)
- [Entropy-based Coarse and Compressed Semantic Speech Representation Learning](https://mlsanity.com/paper/2509.00503/) (2025)
- [Sylber 2.0: A Universal Syllable Embedding](https://mlsanity.com/paper/2601.22306/) (2026)
- [Multilingual Phonological Feature Recognition with Self-Supervised Speech Models](https://mlsanity.com/paper/2605.25596/) (2026)
- [Speech Emotion Recognition with Phonation Excitation Information and Articulatory Kinematics](https://mlsanity.com/paper/2511.07955/) (2025)
- [Anatomy-aware cross-speaker adaptation of complete vocal-tract acoustic-to-articulatory inversion](https://mlsanity.com/paper/2609.29766/) (2026)

## Abstract (arXiv)

Self-Supervised Learning (SSL) based models of speech have shown remarkable performance on a range of downstream tasks. These state-of-the-art models have remained blackboxes, but many recent studies have begun "probing" models like HuBERT, to correlate their internal representations to different aspects of speech. In this paper, we show "inference of articulatory kinematics" as fundamental property of SSL models, i.e., the ability of these models to transform acoustics into the causal articulatory dynamics underlying the speech signal. We also show that this abstraction is largely overlapping across the language of the data used to train the model, with preference to the language with similar phonological system. Furthermore, we show that with simple affine transformations, Acoustic-to-Articulatory inversion (AAI) is transferrable across speakers, even across genders, languages, and dialects, showing the generalizability of this property. Together, these results shed new light on the internals of SSL models that are critical to their superior performance, and open up new avenues into language-agnostic universal models for speech engineering, that are interpretable and grounded in speech science.