# SSR-Speech: Towards Stable, Safe and Robust Zero-shot Text-based Speech Editing and Synthesis

arXiv: https://arxiv.org/abs/2409.07556 · HTML: https://mlsanity.com/paper/2409.07556/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 11 Sep 2024.

## Built on by (most cited first, 13 of 13)

- [Audio Large Language Models Can Be Descriptive Speech Quality Evaluators](https://mlsanity.com/paper/2501.17202/) (2025)
- [CapSpeech: Enabling Downstream Applications in Style-Captioned Text-to-Speech](https://mlsanity.com/paper/2506.02863/) (2025)
- [Deepfake Media Generation and Detection in the Generative AI Era: A Survey and Outlook](https://mlsanity.com/paper/2411.19537/) (2024)
- [OpenOmni: Advancing Open-Source Omnimodal Large Language Models with Progressive Multimodal Alignment and Real-Time Self-Aware Emotional Speech Synthesis](https://mlsanity.com/paper/2501.04561/) (2025)
- [Vevo2: A Unified and Controllable Framework for Speech and Singing Voice Generation](https://mlsanity.com/paper/2508.16332/) (2025)
- [MMAE: A Massive Multitask Audio Editing Benchmark](https://mlsanity.com/paper/2606.07229/) (2026)
- [VoXtream2: Full-stream TTS with dynamic speaking rate control](https://mlsanity.com/paper/2603.13518/) (2026)
- [Deepfake Word Detection by Next-token Prediction using Fine-tuned Whisper](https://mlsanity.com/paper/2602.22658/) (2026)
- [Unifying Speech Editing Detection and Content Localization via Prior-Enhanced Audio LLMs](https://mlsanity.com/paper/2601.21463/) (2026)
- [AST: Adaptive, Seamless, and Training-Free Precise Speech Editing](https://mlsanity.com/paper/2604.16056/) (2026)
- [UniSAE: Unified Speech Attribute Editing on Speaker, Emotion and Low-Level Content via Discrete Phonetic Posteriorgram Modelling](https://mlsanity.com/paper/2606.31128/) (2026)
- [Multi Codec Discrete Diffusion Model for Text Guided Speech Inpainting and Editing](https://mlsanity.com/paper/2608.06424/) (2026)
- [Fish Audio Drama 3 Technical Report](https://mlsanity.com/paper/2610.18010/) (2026)

## Abstract (arXiv)

In this paper, we introduce SSR-Speech, a neural codec autoregressive model designed for stable, safe, and robust zero-shot textbased speech editing and text-to-speech synthesis. SSR-Speech is built on a Transformer decoder and incorporates classifier-free guidance to enhance the stability of the generation process. A watermark Encodec is proposed to embed frame-level watermarks into the edited regions of the speech so that which parts were edited can be detected. In addition, the waveform reconstruction leverages the original unedited speech segments, providing superior recovery compared to the Encodec model. Our approach achieves state-of-the-art performance in the RealEdit speech editing task and the LibriTTS text-to-speech task, surpassing previous methods. Furthermore, SSR-Speech excels in multi-span speech editing and also demonstrates remarkable robustness to background sounds. The source code and demos are released.