# RIVET: Robust Idempotent Voice Attribute Editing

arXiv: https://arxiv.org/abs/2606.19629 · HTML: https://mlsanity.com/paper/2606.19629/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 28 of its 31 references that are in the map.
- First posted on arXiv 17 Jun 2026.

## Builds on (most cited first, 28 of 28)

- [Scaling Rectified Flow Transformers for High-Resolution Image Synthesis](https://mlsanity.com/paper/2403.03206/) (2024)
- [Robust Speech Recognition via Large-Scale Weak Supervision](https://mlsanity.com/paper/2212.04356/) (2022)
- [Consistency Models](https://mlsanity.com/paper/2303.01469/) (2023)
- [ECAPA-TDNN: Emphasized Channel Attention, Propagation and Aggregation in TDNN Based Speaker Verification](https://mlsanity.com/paper/2005.07143/) (2020)
- [Conditional Variational Autoencoder with Adversarial Learning for End-to-End Text-to-Speech](https://mlsanity.com/paper/2106.06103/) (2021)
- [UTMOS: UTokyo-SaruLab System for VoiceMOS Challenge 2022](https://mlsanity.com/paper/2204.02152/) (2022)
- [Understanding disentangling in β-VAE](https://mlsanity.com/paper/1804.03599/) (2018)
- [Flow++: Improving Flow-Based Generative Models with Variational Dequantization and Architecture Design](https://mlsanity.com/paper/1902.00275/) (2019)
- [Semantic Image Inversion and Editing using Rectified Stochastic Differential Equations](https://mlsanity.com/paper/2410.10792/) (2024)
- [DisenBooth: Identity-Preserving Disentangled Tuning for Subject-Driven Text-to-Image Generation](https://mlsanity.com/paper/2305.03374/) (2023)
- [EARS: An Anechoic Fullband Speech Dataset Benchmarked for Speech Enhancement and Dereverberation](https://mlsanity.com/paper/2406.06185/) (2024)
- [TitaNet: Neural Model for speaker representation with 1D Depth-wise separable convolutions and global context](https://mlsanity.com/paper/2110.04410/) (2021)
- [Learning with Neighbor Consistency for Noisy Labels](https://mlsanity.com/paper/2202.02200/) (2022)
- [GLOBE: A High-quality English Corpus with Global Accents for Zero-shot Speaker Adaptive Text-to-Speech](https://mlsanity.com/paper/2406.14875/) (2024)
- [FlashSpeech: Efficient Zero-Shot Speech Synthesis](https://mlsanity.com/paper/2404.14700/) (2024)
- [Label-Noise Robust Generative Adversarial Networks](https://mlsanity.com/paper/1811.11165/) (2018)
- [Consistency Regularization Can Improve Robustness to Label Noise](https://mlsanity.com/paper/2110.01242/) (2021)
- [Idempotent Generative Network](https://mlsanity.com/paper/2311.01462/) (2023)
- [Label-Noise Robust Diffusion Models](https://mlsanity.com/paper/2402.17517/) (2024)
- [Don't drop your samples! Coherence-aware training benefits Conditional diffusion](https://mlsanity.com/paper/2405.20324/) (2024)
- [Imprecise Label Learning: A Unified Framework for Learning with Various Imprecise Label Configurations](https://mlsanity.com/paper/2305.12715/) (2023)
- [Voice Attribute Editing with Text Prompt](https://mlsanity.com/paper/2404.08857/) (2024)
- [VoiceShop: A Unified Speech-to-Speech Framework for Identity-Preserving Zero-Shot Voice Editing](https://mlsanity.com/paper/2404.06674/) (2024)
- [IT3: Idempotent Test-Time Training](https://mlsanity.com/paper/2410.04201/) (2024)
- [Semantic Unfolding of StyleGAN Latent Space](https://mlsanity.com/paper/2206.14892/) (2022)
- [VoxGenesis: Unsupervised Discovery of Latent Speaker Manifold for Speech Synthesis](https://mlsanity.com/paper/2403.00529/) (2024)
- [Guiding Noisy Label Conditional Diffusion Models with Score-based Discriminator Correction](https://mlsanity.com/paper/2508.19581/) (2025)
- [Score-based Idempotent Distillation of Diffusion Models](https://mlsanity.com/paper/2509.21470/) (2025)

## Abstract (arXiv)

Voice attribute editing models modify characteristics such as age and gender while preserving speaker identity. In large-scale speech datasets, however, attribute annotations are often noisy or inconsistent, which can cause conditional generative models to produce unstable edits. In this work, we show that idempotency provides an effective mechanism for improving robustness to noisy labels. An idempotent operator is one for which repeated application does not change the result, i.e., f(f(x)) = f(x). Enforcing this property acts as an implicit regularizer that reduces sensitivity to mislabeled examples. We introduce RIVET, a training framework that incorporates an idempotency objective to improve robustness to label noise. We evaluate RIVET under controlled label noise and on the GLOBE dataset with naturally noisy annotations. RIVET improves editing success and better preserves speaker identity than standard training, showing that idempotency improves robustness in voice editing models.