# Attention-based Interactive Disentangling Network for Instance-level Emotional Voice Conversion

arXiv: https://arxiv.org/abs/2312.17508 · HTML: https://mlsanity.com/paper/2312.17508/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 19 of its 30 references that are in the map.
- First posted on arXiv 29 Dec 2023.

## Builds on (most cited first, 19 of 19)

- [Perceptual Losses for Real-Time Style Transfer and Super-Resolution](https://mlsanity.com/paper/1603.08155/) (2016)
- [Learning Deep Features for Discriminative Localization](https://mlsanity.com/paper/1512.04150/) (2015)
- [HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesis](https://mlsanity.com/paper/2010.05646/) (2020)
- [AUTOVC: Zero-Shot Voice Style Transfer with Only Autoencoder Loss](https://mlsanity.com/paper/1905.05879/) (2019)
- [Seen and Unseen emotional style transfer for voice conversion with a new emotional speech dataset](https://mlsanity.com/paper/2010.14794/) (2020)
- [Emotional Voice Conversion: Theory, Databases and ESD](https://mlsanity.com/paper/2105.14762/) (2021)
- [Self-supervised Image-specific Prototype Exploration for Weakly Supervised Semantic Segmentation](https://mlsanity.com/paper/2203.02909/) (2022)
- [Exploring Dual-task Correlation for Pose Guided Person Image Generation](https://mlsanity.com/paper/2203.02910/) (2022)
- [StarGANv2-VC: A Diverse, Unsupervised, Non-parallel Framework for Natural-Sounding Voice Conversion](https://mlsanity.com/paper/2107.10394/) (2021)
- [Emotion Intensity and its Control for Emotional Voice Conversion](https://mlsanity.com/paper/2201.03967/) (2022)
- [Transforming Spectrum and Prosody for Emotional Voice Conversion with Non-Parallel Training Data](https://mlsanity.com/paper/2002.00198/) (2020)
- [V2C: Visual Voice Cloning](https://mlsanity.com/paper/2111.12890/) (2021)
- [Converting Anyone's Emotion: Towards Speaker-Independent Emotional Voice Conversion](https://mlsanity.com/paper/2005.07025/) (2020)
- [Cross-speaker emotion disentangling and transfer for end-to-end speech synthesis](https://mlsanity.com/paper/2109.06733/) (2021)
- [VAW-GAN for Disentanglement and Recomposition of Emotional Elements in Speech](https://mlsanity.com/paper/2011.02314/) (2020)
- [Nonparallel Emotional Speech Conversion](https://mlsanity.com/paper/1811.01174/) (2018)
- [Limited Data Emotional Voice Conversion Leveraging Text-to-Speech: Two-stage Sequence-to-Sequence Training](https://mlsanity.com/paper/2103.16809/) (2021)
- [Emotional Voice Conversion using Multitask Learning with Text-to-speech](https://mlsanity.com/paper/1911.06149/) (2019)
- [Non-parallel Emotion Conversion using a Deep-Generative Hybrid Network and an Adversarial Pair Discriminator](https://mlsanity.com/paper/2007.12932/) (2020)

## Built on by (most cited first, 3 of 3)

- [A Review of Human Emotion Synthesis Based on Generative Technology](https://mlsanity.com/paper/2412.07116/) (2024)
- [ClapFM-EVC: High-Fidelity and Flexible Emotional Voice Conversion with Dual Control from Natural Language and Speech](https://mlsanity.com/paper/2505.13805/) (2025)
- [Zero Shot Audio to Audio Emotion Transfer With Speaker Disentanglement](https://mlsanity.com/paper/2401.04511/) (2024)

## Abstract (arXiv)

Emotional Voice Conversion aims to manipulate a speech according to a given emotion while preserving non-emotion components. Existing approaches cannot well express fine-grained emotional attributes. In this paper, we propose an Attention-based Interactive diseNtangling Network (AINN) that leverages instance-wise emotional knowledge for voice conversion. We introduce a two-stage pipeline to effectively train our network: Stage I utilizes inter-speech contrastive learning to model fine-grained emotion and intra-speech disentanglement learning to better separate emotion and content. In Stage II, we propose to regularize the conversion with a multi-view consistency mechanism. This technique helps us transfer fine-grained emotion and maintain speech content. Extensive experiments show that our AINN outperforms state-of-the-arts in both objective and subjective metrics.