mlsanityData to 14 Oct 2026

Data-efficient Targeted Token-level Preference Optimization for LLM-based Text-to-Speech

arXiv 2510.05799 ↗First posted 7 Oct 2025cs.CL cs.AI cs.SDcited by 0

On the map: LLM alignment / RLHF / preference optimisation → Preference alignment for recommenders and RAG

Abstract (arXiv)

Aligning text-to-speech (TTS) system outputs with human feedback through preference optimization has been shown to effectively improve the robustness and naturalness of language model-based TTS models. Current approaches primarily require paired desirable and undesirable samples at the utterance level. However, such pairs are often limited in TTS output data, and utterance-level formulation prevents fine-grained token-level optimization needed for accurate pronunciation alignment. In this study, we propose TKTO that eliminates the need for paired data, enabling a more data-efficient training paradigm, and directly targets token-level units, automatically providing fine-grained alignment signals without token-level annotations. TKTO improves the challenging Japanese TTS accuracy by 39% and reduces CER by 54%, automatically assigning 12.8 times stronger reward to targeted tokens.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Preference alignment for recommenders and RAG, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 12 of its 18 references that are in the map. First posted on arXiv 7 Oct 2025.

Builds on

12 of 18 references.

Closest ancestors · 2

Same direction, newest first

  1. 2024Preference Alignment Improves Language Model-Based TTS19 Sep
  2. 2024SpeechAlign: Aligning Speech Generation to Human Preferences8 Apr

Other references · 10

  1. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  2. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 4,057
  3. 2024F5-TTS: A Fairytaler that Fakes Fluent and Faithful Speech with Flow Matchingcited by 361
  4. 2024From r to Q^*: Your Language Model is Secretly a Q-Functioncited by 242
  5. 2024MaskGCT: Zero-Shot Text-to-Speech with Masked Generative Codec Transformercited by 161
  6. 2024Autoregressive Speech Synthesis without Vector Quantizationcited by 65
  7. 2024TIS-DPO: Token-level Importance Sampling for Direct Preference Optimization With Estimated Weightscited by 42
  8. 2025Advancing Zero-shot Text-to-Speech Intelligibility across Diverse Domains via Preference Alignmentcited by 18
  9. 2024CM-TTS: Enhancing Real Time Text-to-Speech Synthesis Efficiency through Weighted Samplers and Consistency Modelscited by 13
  10. 2024HALL-E: Hierarchical Neural Codec Language Model for Minute-Long Zero-Shot Text-to-Speech Synthesiscited by 3

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Speech-Worthy Alignment for Japanese SpeechLLMs via Direct Preference Optimization
  2. 2026Ready to Speak: Aligning LLMs for TTS-Friendly Text Generation
  3. 2025Align2Speak: Improving TTS for Low Resource Languages via ASR-Guided Online Preference Optimization
  4. 2026CTC-TTS: LLM-based dual-streaming text-to-speech with CTC alignment
  5. 2025Koel-TTS: Enhancing LLM based Speech Generation with Preference Alignment and Classifier Free Guidance
  6. 2025LatinX: Aligning a Multilingual TTS Model with Direct Preference Optimization
  7. 2025DiffPO: Diffusion-styled Preference Optimization for Efficient Inference-Time Alignment of Large Language Models
  8. 2025No Verifiable Reward for Prosody: Toward Preference-Guided Prosody Learning in TTS