mlsanityData to 14 Oct 2026

Direct Preference Optimization for Speech Autoregressive Diffusion Models

arXiv 2509.18928 ↗First posted 23 Sep 2025eess.AScited by 4

On the map: Diffusion models → Audio, speech and music

Abstract (arXiv)

Autoregressive diffusion models (ARDMs) have recently been applied to speech generation, achieving state-of-the-art (SOTA) performance in zero-shot text-to-speech. By autoregressively generating continuous speech tokens with next-token diffusion, these models offer a promising alternative to next-token prediction, avoiding the technical complexities associated with discrete speech tokenization. As a relatively new paradigm, research on reinforcement learning (RL)-based fine-tuning of speech ARDMs remains limited. In this paper, we propose Autoregressive Diffusion-Direct Preference Optimization (ARDM-DPO) to advance this research. By fine-tuning the recently proposed zero-shot text-to-speech model DiTAR with DPO, we achieve significant improvements in terms of speech expressiveness and robustness for long texts.

Key facts. Cited by 4 papers in the map (data through 14 Oct 2026). Placed in Audio, speech and music, one of 14 directions in Diffusion models. Builds on 39 of its 40 references that are in the map. First posted on arXiv 23 Sep 2025.

Builds on

39 of 40 references.

Closest ancestors · 8

Same direction, newest first

  1. 2025Fine-Tuning Text-to-Speech Diffusion Models Using Reinforcement Learning with Human Feedback5 Aug
  2. 2025Next Tokens Denoising for Speech Synthesis30 Jul
  3. 2025SongBloom: Coherent Song Generation via Interleaved Autoregressive Sketching and Diffusion Refinement9 Jun
  4. 2025CosyVoice 3: Towards In-the-wild Speech Generation via Scaling-up and Post-training23 May
  5. 2025DiTAR: Diffusion Transformer Autoregressive Modeling for Speech Generation6 Feb
  6. 2024DMOSpeech: Direct Metric Optimization via Distilled Diffusion Model in Zero-Shot Speech Synthesis14 Oct
  7. 2024Autoregressive Diffusion Transformer for Text-to-Speech Synthesis8 Jun
  8. 2024Tango 2: Aligning Diffusion-based Text-to-Audio Generations through Direct Preference Optimization15 Apr

Other references · 31

  1. 2020Denoising Diffusion Probabilistic Modelscited by 21,076
  2. 2022Training language models to follow instructions with human feedbackcited by 15,971
  3. 2020Denoising Diffusion Implicit Modelscited by 9,781
  4. 2020Score-Based Generative Modeling through Stochastic Differential Equationscited by 9,122
  5. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  6. 2023ImageReward: Learning and Evaluating Human Preferences for Text-to-Image Generationcited by 1,045
  7. 2023Diffusion Model Alignment Using Direct Preference Optimizationcited by 790
  8. 2019Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learningcited by 782
  9. 2023RAFT: Reward rAnked FineTuning for Generative Foundation Model Alignmentcited by 686
  10. 2024Autoregressive Image Generation without Vector Quantizationcited by 651
  11. 2019LibriTTS: A Corpus Derived from LibriSpeech for Text-to-Speechcited by 649
  12. 2024From Slow Bidirectional to Fast Autoregressive Video Diffusion Modelscited by 433
19 more
  1. 2024Seed-TTS: A Family of High-Quality Versatile Speech Generation Modelscited by 300
  2. 2025MAGI-1: Autoregressive Video Generation at Scalecited by 276
  3. 2024Autoregressive Video Generation without Vector Quantizationcited by 186
  4. 2025NextStep-1: Toward Autoregressive Image Generation with Continuous Tokens at Scalecited by 57
  5. 2024Multimodal Latent Language Modeling with Next-Token Diffusioncited by 48
  6. 2024SpeechAlign: Aligning Speech Generation to Human Preferencescited by 47
  7. 2024Orthus: Autoregressive Interleaved Image-Text Generation with Modality-Specific Headscited by 44
  8. 2025Recent Advances in Discrete Speech Tokens: A Reviewcited by 43
  9. 2020DiDiSpeech: A Large Scale Mandarin Speech Corpuscited by 33
  10. 2024Emo-DPO: Controllable Emotional Speech Synthesis through Direct Preference Optimizationcited by 32
  11. 2025VibeVoice Technical Reportcited by 29
  12. 20243D-Properties: Identifying Challenges in DPO and Charting a Path Forwardcited by 24
  13. 2020DiscreTalk: Text-to-Speech as a Machine Translation Problemcited by 21
  14. 2024Preference Alignment Improves Language Model-Based TTScited by 21
  15. 2024Enhancing Zero-shot Text-to-Speech Synthesis with Human Feedbackcited by 20
  16. 2025Advancing Zero-shot Text-to-Speech Intelligibility across Diverse Domains via Preference Alignmentcited by 18
  17. 2024Robust Zero-Shot Text-to-Speech Synthesis with Reverse Inference Optimizationcited by 11
  18. 2025Fine-grained Preference Optimization Improves Zero-shot Text-to-Speechcited by 4
  19. 2025MPO: Multidimensional Preference Optimization for Language Model-based Text-to-Speechcited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 4.

4 papers.

  1. 2026Do Audio LLMs Listen or Read? Analyzing and Mitigating Paralinguistic Failures with VoxParadoxcited by 7
  2. 2026GROW: Group-Relative Advantage-Weighted On-Policy Reinforcement Learning of Autoregressive-Diffusion Text-to-Speech modelcited by 1
  3. 2026Bridging the Stability-Expressivity Gap: Synthetic Data Scaling and Preference Alignment for Low-Resource Spoken Language Modelscited by 0
  4. 2026NVAlign: Direct-Gradient Optimization for Non-Verbal Control in Continuous Autoregressive Flow Matching Text-to-Speechcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026DELTA-TTS: Adapting Autoregressive Model into Diffusion Language Model for Text-to-Speech
  2. 2025DMOSpeech 2: Reinforcement Learning for Duration Prediction in Metric-Optimized Speech Synthesis
  3. 2026DiTAR+: Dual Optimization for Robust Autoregressive Diffusion Speech Synthesis
  4. 2023Minimally-Supervised Speech Synthesis with Conditional Diffusion Model and Language Model: A Comparative Study of Semantic Coding
  5. 2025Explore the Reinforcement Learning for the LLM based ASR and TTS system
  6. 2024DLPO: Diffusion Model Loss-Guided Reinforcement Learning for Fine-Tuning Text-to-Speech Diffusion Models
  7. 2021Diff-TTS: A Denoising Diffusion Model for Text-to-Speech
  8. 2025DiSTAR: Diffusion over a Scalable Token Autoregressive Representation for Speech Generation