Residual Tokens Enhance Masked Autoencoders for Speech Modeling
Not on the map. Why?
Abstract (arXiv)
Recent speech modeling relies on explicit attributes such as pitch, content, and speaker identity, but these alone cannot capture the full richness of natural speech. We introduce RT-MAE, a novel masked autoencoder framework that augments the supervised attributes-based modeling with unsupervised residual trainable tokens, designed to encode the information not explained by explicit labeled factors (e.g., timbre variations, noise, emotion etc). Experiments show that RT-MAE improves reconstruction quality, preserving content and speaker similarity while enhancing expressivity. We further demonstrate its applicability to speech enhancement, removing noise at inference while maintaining controllability and naturalness.
Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 28 of its 41 references that are in the map. First posted on arXiv 27 Jan 2026.
Builds on
28 of 41 references.
References · 28
- 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,403
- 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,342
- 2020HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesiscited by 1,154
- 2021SoundStream: An End-to-End Neural Audio Codeccited by 771
- 2022High Fidelity Neural Audio Compressioncited by 687
- 2021Perceiver IO: A General Architecture for Structured Inputs & Outputscited by 629
- 2020ECAPA-TDNN: Emphasized Channel Attention, Propagation and Aggregation in TDNN Based Speaker Verificationcited by 628
- 2018Conv-TasNet: Surpassing Ideal Time-Frequency Magnitude Masking for Speech Separationcited by 406
- 2017Generalized End-to-End Loss for Speaker Verificationcited by 293
- 2018Style Tokens: Unsupervised Style Modeling, Control and Transfer in End-to-End Speech Synthesiscited by 259
- 2021Speech Resynthesis from Discrete Disentangled Self-Supervised Representationscited by 243
- 2020CLUB: A Contrastive Log-ratio Upper Bound of Mutual Informationcited by 235
16 more
- 2019AUTOVC: Zero-Shot Voice Style Transfer with Only Autoencoder Losscited by 192
- 2021DNSMOS P.835: A Non-Intrusive Perceptual Objective Speech Quality Metric to Evaluate Noise Suppressorscited by 172
- 2019WHAM!: Extending Speech Separation to Noisy Environmentscited by 156
- 2020LibriMix: An Open-Source Dataset for Generalizable Speech Separationcited by 155
- 2020DCCRN: Deep Complex Convolution Recurrent Network for Phase-Aware Speech Enhancementcited by 141
- 2024Spirit LM: Interleaved Spoken and Written Language Modelcited by 126
- 2018CREPE: A Convolutional Representation for Pitch Estimationcited by 110
- 2017Unsupervised Learning of Disentangled and Interpretable Representations from Sequential Datacited by 93
- 2018Hierarchical Generative Modeling for Controllable Speech Synthesiscited by 93
- 2021VQMIVC: Vector Quantization and Mutual Information-Based Unsupervised Speech Representation Disentanglement for One-shot Voice Conversioncited by 64
- 2018The Emotional Voices Database: Towards Controlling the Emotion Dimension in Voice Generation Systemscited by 61
- 2023TorchAudio-Squim: Reference-less Speech Quality and Intelligibility measures in TorchAudiocited by 59
- 2024SpeechBERTScore: Reference-Aware Automatic Evaluation of Speech Generation Leveraging NLP Evaluation Metricscited by 38
- 2019StarGAN-VC2: Rethinking Conditional Methods for StarGAN-Based Voice Conversioncited by 37
- 2024High-Fidelity Neural Phonetic Posteriorgramscited by 9
- 2025AnCoGen: Analysis, Control and Generation of Speech with a Masked Autoencodercited by 3
Built on by
Citing papers per year: 2026: 1.
1 paper.