# Articulation-Informed ASR: Integrating Articulatory Features into ASR via Auxiliary Speech Inversion and Cross-Attention Fusion

arXiv: https://arxiv.org/abs/2510.08585 · HTML: https://mlsanity.com/paper/2510.08585/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 11 of its 19 references that are in the map.
- First posted on arXiv 1 Oct 2025.

## Builds on (most cited first, 11 of 11)

- [Robust Speech Recognition via Large-Scale Weak Supervision](https://mlsanity.com/paper/2212.04356/) (2022)
- [wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations](https://mlsanity.com/paper/2006.11477/) (2020)
- [Multi-Task Learning Using Uncertainty to Weigh Losses for Scene Geometry and Semantics](https://mlsanity.com/paper/1705.07115/) (2017)
- [Multimodal Approach for Assessing Neuromotor Coordination in Schizophrenia Using Convolutional Neural Networks](https://mlsanity.com/paper/2110.04440/) (2021)
- [Speech-Based Estimation of Schizophrenia Severity Using Feature Fusion](https://mlsanity.com/paper/2411.06033/) (2024)
- [Multimodal Biomarkers for Schizophrenia: Towards Individual Symptom Severity Estimation](https://mlsanity.com/paper/2505.16044/) (2025)
- [Acoustic-to-articulatory Speech Inversion with Multi-task Learning](https://mlsanity.com/paper/2205.13755/) (2022)
- [Enhancing Acoustic-to-Articulatory Speech Inversion by Incorporating Nasality](https://mlsanity.com/paper/2506.09231/) (2025)
- [Inverted Vocal Tract Variables and Facial Action Units to Quantify Neuromotor Coordination in Schizophrenia](https://mlsanity.com/paper/2102.07054/) (2021)
- [Acoustic To Articulatory Speech Inversion Using Multi-Resolution Spectro-Temporal Representations Of Speech Signals](https://mlsanity.com/paper/2203.05780/) (2022)
- [Perceptual Ratings Predict Speech Inversion Articulatory Kinematics in Childhood Speech Sound Disorders](https://mlsanity.com/paper/2507.01888/) (2025)

## Abstract (arXiv)

Prior works have investigated the use of articulatory features as complementary representations for automatic speech recognition (ASR), but their use was largely confined to shallow acoustic models. In this work, we revisit articulatory information in the era of deep learning and propose a framework that leverages articulatory representations both as an auxiliary task and as a pseudo-input to the recognition model. Specifically, we employ speech inversion as an auxiliary prediction task, and the predicted articulatory features are injected into the model as a query stream in a cross-attention module with acoustic embeddings as keys and values. Experiments on LibriSpeech demonstrate that our approach yields consistent improvements over strong transformer-based baselines, particularly under low-resource conditions. These findings suggest that articulatory features, once sidelined in ASR research, can provide meaningful benefits when reintroduced with modern architectures.