mlsanityData to 7 Oct 2026

DPI-TTS: Directional Patch Interaction for Fast-Converging and Style Temporal Modeling in Text-to-Speech

arXiv 2409.11835 ↗First posted 18 Sep 2024cs.SD cs.AI eess.AScited by 1

On the map: Diffusion models → Audio, speech and music

Abstract (arXiv)

In recent years, speech diffusion models have advanced rapidly. Alongside the widely used U-Net architecture, transformer-based models such as the Diffusion Transformer (DiT) have also gained attention. However, current DiT speech models treat Mel spectrograms as general images, which overlooks the specific acoustic properties of speech. To address these limitations, we propose a method called Directional Patch Interaction for Text-to-Speech (DPI-TTS), which builds on DiT and achieves fast training without compromising accuracy. Notably, DPI-TTS employs a low-to-high frequency, frame-by-frame progressive inference approach that aligns more closely with acoustic properties, enhancing the naturalness of the generated speech. Additionally, we introduce a fine-grained style temporal modeling method that further improves speaker style similarity. Experimental results demonstrate that our method increases the training speed by nearly 2 times and significantly outperforms the baseline models.

Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Placed in Audio, speech and music, one of 14 directions in Diffusion models. Builds on 26 of its 31 references that are in the map. First posted on arXiv 18 Sep 2024.

Builds on

26 of 31 references.

Closest ancestors · 11

Same direction, newest first

  1. 2023U-DiT TTS: U-Diffusion Vision Transformer for Text-to-Speech22 May
  2. 2023CoMoSpeech: One-Step Speech and Singing Voice Synthesis via Consistency Model11 May
  3. 2023NaturalSpeech 2: Latent Diffusion Models are Natural and Zero-Shot Speech and Singing Synthesizers18 Apr
  4. 2022ProDiff: Progressive Fast Diffusion Model For High-Quality Text-to-Speech13 Jul
  5. 2022FastDiff: A Fast Conditional Diffusion Model for High-Quality Speech Synthesis21 Apr
  6. 2022SpecGrad: Diffusion Probabilistic Model based Neural Vocoder with Adaptive Noise Spectral Shaping31 Mar
  7. 2022InferGrad: Improving Diffusion Models for Vocoder by Considering Inference in Training8 Feb
  8. 2021PriorGrad: Improving Conditional Denoising Diffusion Models with Data-Dependent Adaptive Prior11 Jun
  9. 2021Grad-TTS: A Diffusion Probabilistic Model for Text-to-Speech13 May
  10. 2021Diff-TTS: A Denoising Diffusion Model for Text-to-Speech3 Apr
  11. 2020DiffWave: A Versatile Diffusion Model for Audio Synthesis21 Sep

Other references · 15

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,604
  3. 2022Scalable Diffusion Models with Transformerscited by 5,931
  4. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,581
  5. 2015Variational Inference with Normalizing Flowscited by 2,161
  6. 2017Natural TTS Synthesis by Conditioning WaveNet on Mel Spectrogram Predictionscited by 800
  7. 2020FastSpeech 2: Fast and High-Quality End-to-End Text to Speechcited by 710
  8. 2021Conditional Variational Autoencoder with Adversarial Learning for End-to-End Text-to-Speechcited by 555
  9. 2022NaturalSpeech: End-to-End Text to Speech Synthesis with Human-Level Qualitycited by 151
  10. 2024ConvConcatNet: a deep convolutional neural network to reconstruct mel spectrogram from the EEGcited by 12
  11. 2023Diffusion-Based Mel-Spectrogram Enhancement for Personalized Speech Synthesis with Found Datacited by 5
  12. 2024DEX-TTS: Diffusion-based EXpressive Text-to-Speech with Style Modeling on Time Variabilitycited by 5
3 more
  1. 2022cMelGAN: An Efficient Conditional Generative Model Based on Mel Spectrogramscited by 2
  2. 2024Mel-FullSubNet: Mel-Spectrogram Enhancement for Improving Both Speech Quality and ASRcited by 1
  3. 2024A Mel Spectrogram Enhancement Paradigm Based on CWT in Speech Synthesiscited by 1

Built on by

Cited by year

Citing papers per year: 2025: 1.

1 paper.

  1. 2025InstructAudio: Unified speech and music generation with natural language instructioncited by 5

Similar, not cited

Close in wording, no citation either way

  1. 2024DiTTo-TTS: Diffusion Transformers for Scalable Text-to-Speech without Domain-Specific Factors
  2. 2025DiTAR: Diffusion Transformer Autoregressive Modeling for Speech Generation
  3. 2024SimpleSpeech 2: Towards Simple and Efficient Text-to-Speech with Flow-based Scalar Latent Transformer Diffusion Models
  4. 2024On the Semantic Latent Space of Diffusion-Based Text-to-Speech Models
  5. 2024StyleTTS-ZS: Efficient High-Quality Zero-Shot Text-to-Speech Synthesis with Distilled Time-Varying Style Diffusion
  6. 2022NoreSpeech: Knowledge Distillation based Conditional Diffusion Model for Noise-robust Expressive TTS
  7. 2025From Text to Talk: Audio-Language Model Needs Non-Autoregressive Joint Training
  8. 2025DMP-TTS: Disentangled multi-modal Prompting for Controllable Text-to-Speech with Chained Guidance