# DurIAN: Duration Informed Attention Network For Multimodal Synthesis

arXiv: https://arxiv.org/abs/1909.01700 · HTML: https://mlsanity.com/paper/1909.01700/

## Key facts

- Cited by 33 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 4 Sep 2019.

## Built on by (most cited first, 30 of 33)

- [RAVE: A variational autoencoder for fast and high-quality neural audio synthesis](https://mlsanity.com/paper/2111.05011/) (2021)
- [WaveGrad 2: Iterative Refinement for Text-to-Speech Synthesis](https://mlsanity.com/paper/2106.09660/) (2021)
- [CoMoSpeech: One-Step Speech and Singing Voice Synthesis via Consistency Model](https://mlsanity.com/paper/2305.06908/) (2023)
- [Transformers in Speech Processing: A Survey](https://mlsanity.com/paper/2303.11607/) (2023)
- [iSTFTNet: Fast and Lightweight Mel-Spectrogram Vocoder Incorporating Inverse Short-Time Fourier Transform](https://mlsanity.com/paper/2203.02395/) (2022)
- [Avocodo: Generative Adversarial Network for Artifact-free Vocoder](https://mlsanity.com/paper/2206.13404/) (2022)
- [Integrated Speech and Gesture Synthesis](https://mlsanity.com/paper/2108.11436/) (2021)
- [Large Language Models Meet Text-Centric Multimodal Sentiment Analysis: A Survey](https://mlsanity.com/paper/2406.08068/) (2024)
- [Diff-TTSG: Denoising probabilistic integrated speech and gesture synthesis](https://mlsanity.com/paper/2306.09417/) (2023)
- [Deep Performer: Score-to-Audio Music Performance Synthesis](https://mlsanity.com/paper/2202.06034/) (2022)
- [Parallel and High-Fidelity Text-to-Lip Generation](https://mlsanity.com/paper/2107.06831/) (2021)
- [Review of end-to-end speech synthesis technology based on deep learning](https://mlsanity.com/paper/2104.09995/) (2021)
- [Unified speech and gesture synthesis using flow matching](https://mlsanity.com/paper/2310.05181/) (2023)
- [Meta-Voice: Fast few-shot style transfer for expressive voice cloning using meta learning](https://mlsanity.com/paper/2111.07218/) (2021)
- [Word-Level Style Control for Expressive, Non-attentive Speech Synthesis](https://mlsanity.com/paper/2111.10173/) (2021)
- [Simple and Effective Multi-sentence TTS with Expressive and Coherent Prosody](https://mlsanity.com/paper/2206.14643/) (2022)
- [Fake it to make it: Using synthetic data to remedy the data shortage in joint multimodal speech-and-gesture synthesis](https://mlsanity.com/paper/2404.19622/) (2024)
- [iSTFTNet2: Faster and More Lightweight iSTFT-Based Neural Vocoder Using 1D-2D CNN](https://mlsanity.com/paper/2308.07117/) (2023)
- [Attention Forcing for Machine Translation](https://mlsanity.com/paper/2104.01264/) (2021)
- [High-Fidelity and Low-Latency Universal Neural Vocoder based on Multiband WaveRNN with Data-Driven Linear Prediction for Discrete Waveform Modeling](https://mlsanity.com/paper/2105.09856/) (2021)
- [Continuous descriptor-based control for deep audio synthesis](https://mlsanity.com/paper/2302.13542/) (2023)
- [Prosody Analysis of Audiobooks](https://mlsanity.com/paper/2310.06930/) (2023)
- [Neural Text to Articulate Talk: Deep Text to Audiovisual Speech Synthesis achieving both Auditory and Photo-realism](https://mlsanity.com/paper/2312.06613/) (2023)
- [Improve GAN-based Neural Vocoder using Pointwise Relativistic LeastSquare GAN](https://mlsanity.com/paper/2103.14245/) (2021)
- [Improving Mandarin Prosodic Structure Prediction with Multi-level Contextual Information](https://mlsanity.com/paper/2308.16577/) (2023)
- [Lightweight End-to-end Text-to-speech Synthesis for low resource on-device applications](https://mlsanity.com/paper/2505.07701/) (2025)
- [Triple M: A Practical Text-to-speech Synthesis System With Multi-guidance Attention And Multi-band Multi-time LPCNet](https://mlsanity.com/paper/2102.00247/) (2021)
- [Multi-Scale Spectrogram Modelling for Neural Text-to-Speech](https://mlsanity.com/paper/2106.15649/) (2021)
- [Referee: Towards reference-free cross-speaker style transfer with low-quality data for expressive speech synthesis](https://mlsanity.com/paper/2109.03439/) (2021)
- [PAMA-TTS: Progression-Aware Monotonic Attention for Stable Seq2Seq TTS With Accurate Phoneme Duration Control](https://mlsanity.com/paper/2110.04486/) (2021)

## Abstract (arXiv)

In this paper, we present a generic and robust multimodal synthesis system that produces highly natural speech and facial expression simultaneously. The key component of this system is the Duration Informed Attention Network (DurIAN), an autoregressive model in which the alignments between the input text and the output acoustic features are inferred from a duration model. This is different from the end-to-end attention mechanism used, and accounts for various unavoidable artifacts, in existing end-to-end speech synthesis systems such as Tacotron. Furthermore, DurIAN can be used to generate high quality facial expression which can be synchronized with generated speech with/without parallel speech and face data. To improve the efficiency of speech generation, we also propose a multi-band parallel generation strategy on top of the WaveRNN model. The proposed Multi-band WaveRNN effectively reduces the total computational complexity from 9.8 to 5.5 GFLOPS, and is able to generate audio that is 6 times faster than real time on a single CPU core. We show that DurIAN could generate highly natural speech that is on par with current state of the art end-to-end systems, while at the same time avoid word skipping/repeating errors in those systems. Finally, a simple yet effective approach for fine-grained control of expressiveness of speech and facial expression is introduced.