mlsanityData to 7 Oct 2026

Improving Speech Inversion Through Self-Supervised Embeddings and Enhanced Tract Variables

arXiv 2309.09220 ↗First posted 17 Sep 2023eess.AS cs.AI cs.SDcited by 6

Not on the map. Why?

Abstract (arXiv)

The performance of deep learning models depends significantly on their capacity to encode input features efficiently and decode them into meaningful outputs. Better input and output representation has the potential to boost models' performance and generalization. In the context of acoustic-to-articulatory speech inversion (SI) systems, we study the impact of utilizing speech representations acquired via self-supervised learning (SSL) models, such as HuBERT compared to conventional acoustic features. Additionally, we investigate the incorporation of novel tract variables (TVs) through an improved geometric transformation model. By combining these two approaches, we improve the Pearson product-moment correlation (PPMC) scores which evaluate the accuracy of TV estimation of the SI system from 0.7452 to 0.8141, a 6.9% increase. Our findings underscore the profound influence of rich feature representations from SSL models and improved geometric transformations with target TVs on the enhanced functionality of SI systems.

Key facts. Cited by 6 papers in the map (data through 7 Oct 2026). Builds on 16 of its 27 references that are in the map. First posted on arXiv 17 Sep 2023.

Builds on

16 of 27 references.

References · 16

  1. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,853
  2. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,301
  3. 2020Unsupervised Cross-lingual Representation Learning for Speech Recognitioncited by 496
  4. 2021SpeechBrain: A General-Purpose Speech Toolkitcited by 404
  5. 2019Mockingjay: Unsupervised Speech Representation Learning with Deep Bidirectional Transformer Encoderscited by 196
  6. 2020TERA: Self-Supervised Learning of Transformer Encoder Representation for Speechcited by 186
  7. 2020Exploring wav2vec 2.0 on speaker verification and language identificationcited by 93
  8. 2022Evidence of Vocal Tract Articulation in Self-Supervised Learning of Speechcited by 30
  9. 2022The Secret Source : Incorporating Source Features to Improve Acoustic-to-Articulatory Speech Inversioncited by 5
  10. 2022An Adapter based Multi-label Pre-training for Speech Separation and Enhancementcited by 5
  11. 2021Multimodal Approach for Assessing Neuromotor Coordination in Schizophrenia Using Convolutional Neural Networkscited by 4
  12. 2022Improved acoustic-to-articulatory inversion using representations from pretrained self-supervised learning modelscited by 4
4 more
  1. 2022Audio Data Augmentation for Acoustic-to-articulatory Speech Inversion using Bidirectional Gated RNNscited by 3
  2. 2022Masked Autoencoders Are Articulatory Learnerscited by 2
  3. 2022Learning to Compute the Articulatory Representations of Speech with the MIRRORNETcited by 2
  4. 2023Enhancing Speech Articulation Analysis using a Geometric Transformation of the X-ray Microbeam Datasetcited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 4; 2026: 1.

6 papers.

  1. 2024Coding Speech through Vocal Tract Kinematicscited by 18
  2. 2025RT-VC: Real-Time Zero-Shot Voice Conversion with Speech Articulatory Codingcited by 4
  3. 2025Multimodal Biomarkers for Schizophrenia: Towards Individual Symptom Severity Estimationcited by 3
  4. 2025Quantifying Articulatory Coordination as a Biomarker for Schizophreniacited by 0
  5. 2025Speech Emotion Recognition with Phonation Excitation Information and Articulatory Kinematicscited by 0
  6. 2026Depression Markers in Speech: An Approach based on Tract Variables Dynamicscited by 0