mlsanityData to 19 Oct 2026

Edit Who Speaks, Control How They Speak: Global Timbre Editing and Local Instruction Control for TTS

arXiv 2610.11437 ↗First posted 8 Oct 2026cs.SD eess.AScited by 0

On the map: Speech and audio language models → Vocoders and speech synthesis

Abstract (arXiv)

Instruction-based text-to-speech (TTS) offers control over voice characteristics and speech expression through interfaces including voice cloning and text-based voice design. Voice cloning reproduces a reference voice, whereas text-based voice design creates a voice from a natural-language description. However, neither interface directly enables users to modify the timbre of a given reference and synthesize speech with the modified voice. Meanwhile, utterance-level expressive instructions leave changes across individual text segments underspecified. We introduce EDICT, a framework that unifies global timbre editing and local expressive control by using an edited acoustic reference to anchor voice identity across segments. To enable synthesis with an instruction-edited voice, EDICT combines reference audio with structured timbre edits to generate an edited reference in codec-token space. This representation serves as a shared voice anchor for a frozen TTS backbone, allowing segment-specific natural-language instructions to guide expression. To accommodate instruction changes while supporting acoustic continuity, EDICT rebuilds the KV cache at each segment boundary, refreshing instruction conditioning while retaining bounded acoustic context from previously generated speech. Evaluations on our proposed TimbreEdit-Bench and IntraTTS-Bench demonstrate improved timbre editing and a favorable balance between local instruction adherence, speaker consistency, and transition quality. Audio demos are available.

Key facts. No citing papers in the map yet (data through 19 Oct 2026). Placed in Vocoders and speech synthesis, one of 11 directions in Speech and audio language models. Builds on 65 of its 69 references that are in the map. First posted on arXiv 8 Oct 2026.

Builds on

65 of 69 references.

Closest ancestors · 32

Same direction, newest first

  1. 2026Diagnose, Then Refine: A Closed-Loop TTS System with AudioLLM-Guided Correction29 Aug
  2. 2026FireRedTTS3: Unified Speech Generation and Editing with Semantically Enriched Speech Representations18 Aug
  3. 2026VoxCPM2 Technical Report5 Jun
  4. 2026FC-TTS: Style and Timbre Control in Zero-Shot Text-to-Speech with Disentangled Speech Representations23 May
  5. 2026MAGIC-TTS: Fine-Grained Controllable Speech Synthesis with Explicit Local Duration and Pause Control23 Apr
  6. 2026Unlocking Fine-Grained and Within-Utterance Speaking Style Control in Prompt-Based Text-to-Speech Models9 Apr
  7. 2026Qwen3-TTS Technical Report22 Jan
  8. 2026A Unified Neural Codec Language Model for Selective Editable Text to Speech Generation18 Jan
  9. 2026VoiceSculptor: Your Voice, Designed By You15 Jan
  10. 2026FlexiVoice: Enabling Flexible Style Control in Zero-Shot TTS with Natural Language Instructions8 Jan
  11. 2026ReStyle-TTS: Relative and Continuous Style Control for Zero-Shot Speech Synthesis7 Jan
  12. 2025Step-Audio-EditX Technical Report5 Nov
20 more
  1. 2025CapSpeech: Enabling Downstream Applications in Style-Captioned Text-to-Speech3 Jun
  2. 2025CosyVoice 3: Towards In-the-wild Speech Generation via Scaling-up and Post-training23 May
  3. 2025Prosody-Adaptable Audio Codecs for Zero-Shot Voice Conversion via In-Context Learning21 May
  4. 2025Spark-TTS: An Efficient LLM-Based Text-to-Speech Model with Single-Stream Decoupled Speech Tokens3 Mar
  5. 2025Koel-TTS: Enhancing LLM based Speech Generation with Preference Alignment and Classifier Free Guidance7 Feb
  6. 2025Speech Synthesis along Perceptual Voice Quality Dimensions15 Jan
  7. 2024CosyVoice 2: Scalable Streaming Speech Synthesis with Large Language Models13 Dec
  8. 2024F5-TTS: A Fairytaler that Fakes Fluent and Faithful Speech with Flow Matching9 Oct
  9. 2024Seed-TTS: A Family of High-Quality Versatile Speech Generation Models4 Jun
  10. 2024ControlSpeech: Towards Simultaneous and Independent Zero-shot Speaker Cloning and Zero-shot Language Style Control3 Jun
  11. 2024NaturalSpeech 3: Zero-Shot Speech Synthesis with Factorized Codec and Diffusion Models5 Mar
  12. 2023PromptTTS++: Controlling Speaker Identity in Prompt-Based Text-to-Speech Using Natural Language Descriptions15 Sep
  13. 2023PromptTTS 2: Describing and Generating Voices with Text Prompt5 Sep
  14. 2023TextrolSpeech: A Text Style Control Speech Corpus With Codec Language Text-to-Speech Models28 Aug
  15. 2023Voicebox: Text-Guided Multilingual Universal Speech Generation at Scale23 Jun
  16. 2023PromptStyle: Controllable Style Transfer for Text-to-Speech with Natural Language Descriptions31 May
  17. 2023InstructTTS: Modelling Expressive TTS in Discrete Latent Space with Natural Language Style Prompt31 Jan
  18. 2023Neural Codec Language Models are Zero-Shot Text to Speech Synthesizers5 Jan
  19. 2022PromptTTS: Controllable Text-to-Speech with Text Descriptions22 Nov
  20. 2021Fine-grained style control in Transformer-based Text-to-speech Synthesis12 Oct

Other references · 33

  1. 2025Qwen3 Technical Reportcited by 9,258
  2. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,210
  3. 2021WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processingcited by 2,135
  4. 2022UTMOS: UTokyo-SaruLab System for VoiceMOS Challenge 2022cited by 552
  5. 2018Style Tokens: Unsupervised Style Modeling, Control and Transfer in End-to-End Speech Synthesiscited by 364
  6. 2021DNSMOS P.835: A Non-Intrusive Perceptual Objective Speech Quality Metric to Evaluate Noise Suppressorscited by 292
  7. 2025LongLive: Real-time Interactive Long Video Generationcited by 276
  8. 2023emotion2vec: Self-Supervised Pre-Training for Speech Emotion Representationcited by 263
  9. 2021Emotional Voice Conversion: Theory, Databases and ESDcited by 146
  10. 2025IndexTTS2: A Breakthrough in Emotionally Expressive and Duration-Controlled Auto-Regressive Zero-Shot Text-to-Speechcited by 142
  11. 2024Provably Mitigating Overoptimization in RLHF: Your SFT Loss is Implicitly an Adversarial Regularizercited by 115
  12. 2022FreeVC: Towards High-Quality Text-Free One-Shot Voice Conversioncited by 105
21 more
  1. 2023An Enhanced Res2Net with Local and Global Feature Fusion for Speaker Verificationcited by 62
  2. 2025EmoVoice: LLM-based Emotional Text-To-Speech Model with Freestyle Text Promptingcited by 58
  3. 2025Vevo: Controllable Zero-Shot Voice Imitation with Self-Supervised Disentanglementcited by 52
  4. 2024VoxInstruct: Expressive Human Instruction-to-Speech Generation with Unified Multilingual Codec Language Modellingcited by 43
  5. 2024SpeechCraft: A Fine-grained Expressive Speech Dataset with Natural Language Descriptioncited by 33
  6. 2025InstructTTSEval: Benchmarking Complex Natural-Language Instruction Following in Text-to-Speech Systemscited by 31
  7. 2024Laugh Now Cry Later: Controlling Time-Varying Emotional States of Flow-Matching-Based Zero-Shot Text-to-Speechcited by 30
  8. 2024VoiceShop: A Unified Speech-to-Speech Framework for Identity-Preserving Zero-Shot Voice Editingcited by 14
  9. 2024Voice Attribute Editing with Text Promptcited by 10
  10. 2024DreamVoice: Text-Guided Voice Conversioncited by 10
  11. 2025Word-Level Emotional Expression Control in Zero-Shot Text-to-Speech Synthesiscited by 9
  12. 2025InstructAudio: Unified speech and music generation with natural language instructioncited by 8
  13. 2025CoMelSinger: Discrete Token-Based Zero-Shot Singing Synthesis With Structured Melody Control and Guidancecited by 7
  14. 2026EmoTra-TTS: Smooth Intra-Utterance Emotion Transitions for Speech Synthesiscited by 6
  15. 2026TED-TTS: Training-Free Intra-Utterance Emotion and Duration Control for Text-to-Speech Synthesiscited by 4
  16. 2026Preference Optimization for Non-Verbal Vocalization Synthesiscited by 4
  17. 2026Sequential Trajectories and Simultaneous Blending: Multi-Emotion Modeling for Instruction-Following TTScited by 3
  18. 2025TTS-CtrlNet: Time varying emotion aligned text-to-speech generation with ControlNetcited by 2
  19. 2026VoiceDesigner: Text-to-Voice Generation and Editing via Unified Diffusion Modeling and Data Augmentationcited by 2
  20. 2025In This Environment, As That Speaker: A Text-Driven Framework for Multi-Attribute Speech Conversioncited by 1
  21. 2026Poly-InstructTTS: Learning In-the-Wild Expressive Speech Synthesis from Open-Ended Instructionscited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026CtrlSpeech: Coarse-to-Fine Control for Expressive Speech Synthesis
  2. 2026dots.tts.edit: Precisely Controlled Speech Editing with a Continuous Autoregressive Model
  3. 2026Harness TTS: Towards Context-Aware Expressive Speech Synthesis with Harness Layer
  4. 2026SCIC: Scope- and Codebook-Aware Instruction Conditioning for Speaker-Adapted Expressive TTS
  5. 2026InstCharVoice: Grounding Natural-Language Instructions for Character-Level Control in Text-to-Speech
  6. 2026CAST-TTS: A Simple Cross-Attention Framework for Unified Timbre Control in TTS
  7. 2025ISSE: An Instruction-Guided Speech Style Editing Dataset And Benchmark
  8. 2025Do You Hear What I Mean? Quantifying the Instruction-Perception Gap in Instruction-Guided Expressive Text-To-Speech Systems