# Improving Speech Inversion Through Self-Supervised Embeddings and Enhanced Tract Variables

arXiv: https://arxiv.org/abs/2309.09220 · HTML: https://mlsanity.com/paper/2309.09220/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 16 of its 27 references that are in the map.
- First posted on arXiv 17 Sep 2023.

## Builds on (most cited first, 16 of 16)

- [wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations](https://mlsanity.com/paper/2006.11477/) (2020)
- [HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Units](https://mlsanity.com/paper/2106.07447/) (2021)
- [Unsupervised Cross-lingual Representation Learning for Speech Recognition](https://mlsanity.com/paper/2006.13979/) (2020)
- [SpeechBrain: A General-Purpose Speech Toolkit](https://mlsanity.com/paper/2106.04624/) (2021)
- [Mockingjay: Unsupervised Speech Representation Learning with Deep Bidirectional Transformer Encoders](https://mlsanity.com/paper/1910.12638/) (2019)
- [TERA: Self-Supervised Learning of Transformer Encoder Representation for Speech](https://mlsanity.com/paper/2007.06028/) (2020)
- [Exploring wav2vec 2.0 on speaker verification and language identification](https://mlsanity.com/paper/2012.06185/) (2020)
- [Evidence of Vocal Tract Articulation in Self-Supervised Learning of Speech](https://mlsanity.com/paper/2210.11723/) (2022)
- [The Secret Source : Incorporating Source Features to Improve Acoustic-to-Articulatory Speech Inversion](https://mlsanity.com/paper/2210.16450/) (2022)
- [An Adapter based Multi-label Pre-training for Speech Separation and Enhancement](https://mlsanity.com/paper/2211.06041/) (2022)
- [Multimodal Approach for Assessing Neuromotor Coordination in Schizophrenia Using Convolutional Neural Networks](https://mlsanity.com/paper/2110.04440/) (2021)
- [Improved acoustic-to-articulatory inversion using representations from pretrained self-supervised learning models](https://mlsanity.com/paper/2210.16871/) (2022)
- [Audio Data Augmentation for Acoustic-to-articulatory Speech Inversion using Bidirectional Gated RNNs](https://mlsanity.com/paper/2205.13086/) (2022)
- [Masked Autoencoders Are Articulatory Learners](https://mlsanity.com/paper/2210.15195/) (2022)
- [Learning to Compute the Articulatory Representations of Speech with the MIRRORNET](https://mlsanity.com/paper/2210.16454/) (2022)
- [Enhancing Speech Articulation Analysis using a Geometric Transformation of the X-ray Microbeam Dataset](https://mlsanity.com/paper/2305.10775/) (2023)

## Built on by (most cited first, 7 of 7)

- [Coding Speech through Vocal Tract Kinematics](https://mlsanity.com/paper/2406.12998/) (2024)
- [RT-VC: Real-Time Zero-Shot Voice Conversion with Speech Articulatory Coding](https://mlsanity.com/paper/2506.10289/) (2025)
- [Multimodal Biomarkers for Schizophrenia: Towards Individual Symptom Severity Estimation](https://mlsanity.com/paper/2505.16044/) (2025)
- [Quantifying Articulatory Coordination as a Biomarker for Schizophrenia](https://mlsanity.com/paper/2511.03084/) (2025)
- [Speech Emotion Recognition with Phonation Excitation Information and Articulatory Kinematics](https://mlsanity.com/paper/2511.07955/) (2025)
- [Depression Markers in Speech: An Approach based on Tract Variables Dynamics](https://mlsanity.com/paper/2607.25888/) (2026)
- [Articulatory Entrainment and Coordination Complexity in Spontaneous Autistic and Non-autistic Dialogue](https://mlsanity.com/paper/2610.04071/) (2026)

## Abstract (arXiv)

The performance of deep learning models depends significantly on their capacity to encode input features efficiently and decode them into meaningful outputs. Better input and output representation has the potential to boost models' performance and generalization. In the context of acoustic-to-articulatory speech inversion (SI) systems, we study the impact of utilizing speech representations acquired via self-supervised learning (SSL) models, such as HuBERT compared to conventional acoustic features. Additionally, we investigate the incorporation of novel tract variables (TVs) through an improved geometric transformation model. By combining these two approaches, we improve the Pearson product-moment correlation (PPMC) scores which evaluate the accuracy of TV estimation of the SI system from 0.7452 to 0.8141, a 6.9% increase. Our findings underscore the profound influence of rich feature representations from SSL models and improved geometric transformations with target TVs on the enhanced functionality of SI systems.