mlsanityData to 19 Oct 2026

STCTS: Generative Semantic Compression for Ultra-Low Bitrate Speech via Explicit Text-Prosody-Timbre Decomposition

arXiv 2512.00451 ↗First posted 29 Nov 2025cs.SD cs.MMcited by 0

On the map: Speech and audio language models → Vocoders and speech synthesis

Abstract (arXiv)

Voice communication in bandwidth-constrained environments--maritime, satellite, and tactical networks--remains prohibitively expensive. Traditional codecs struggle below 1 kbps, while existing semantic approaches (STT-TTS) sacrifice prosody and speaker identity. We present STCTS, a generative semantic compression framework enabling natural voice communication at 80 bps. STCTS explicitly decomposes speech into linguistic content, prosodic expression, and speaker timbre, applying tailored compression: context-aware text encoding (70 bps), sparse prosody transmission via TTS interpolation (<14 bps at 0.1-1 Hz), and amortized speaker embedding. Evaluations on LibriSpeech demonstrate a 75x bitrate reduction versus Opus (6 kbps) and 12x versus EnCodec (1 kbps), while maintaining perceptual quality (NISQA MOS > 4.26), graceful degradation under packet loss and noise resilience. We also discover a bimodal quality distribution with prosody sampling rate: sparse and dense updates both achieve high quality, while mid-range rates degrade due to perceptual discontinuities--guiding optimal configuration design. Beyond efficiency, our modular architecture supports privacy-preserving encryption, human-interpretable transmission, and flexible deployment on edge devices, offering a robust solution for ultra-low bandwidth scenarios.

Key facts. No citing papers in the map yet (data through 19 Oct 2026). Placed in Vocoders and speech synthesis, one of 11 directions in Speech and audio language models. Builds on 23 of its 40 references that are in the map. First posted on arXiv 29 Nov 2025.

Builds on

23 of 40 references.

Closest ancestors · 6

Same direction, newest first

  1. 2025A Novel Semantic Compression Approach for Ultra-low Bandwidth Voice Communication18 Sep
  2. 2024VALL-E 2: Neural Codec Language Models are Human Parity Zero-Shot Text to Speech Synthesizers8 Jun
  3. 2021YourTTS: Towards Zero-Shot Multi-Speaker TTS and Zero-Shot Voice Conversion for everyone4 Dec
  4. 2021SoundStream: An End-to-End Neural Audio Codec7 Jul
  5. 2021A Survey on Neural Speech Synthesis29 Jun
  6. 2021Conditional Variational Autoencoder with Adversarial Learning for End-to-End Text-to-Speech11 Jun

Other references · 17

  1. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 4,661
  2. 2020Conformer: Convolution-augmented Transformer for Speech Recognitioncited by 2,293
  3. 2017Natural TTS Synthesis by Conditioning WaveNet on Mel Spectrogram Predictionscited by 1,023
  4. 2020ECAPA-TDNN: Emphasized Channel Attention, Propagation and Aggregation in TDNN Based Speaker Verificationcited by 980
  5. 2020FastSpeech 2: Fast and High-Quality End-to-End Text to Speechcited by 947
  6. 2022High Fidelity Neural Audio Compressioncited by 917
  7. 2021SpeechBrain: A General-Purpose Speech Toolkitcited by 532
  8. 2021NISQA: A Deep CNN-Self-Attention Model for Multidimensional Speech Quality Prediction with Crowdsourced Datasetscited by 270
  9. 2023Language Modeling Is Compressioncited by 232
  10. 2019AUTOVC: Zero-Shot Voice Style Transfer with Only Autoencoder Losscited by 224
  11. 2018LPCNet: Improving Neural Speech Synthesis Through Linear Predictioncited by 160
  12. 2024SemantiCodec: An Ultra Low Bitrate Semantic Audio Codec for General Soundcited by 84
5 more
  1. 2021VQMIVC: Vector Quantization and Mutual Information-Based Unsupervised Speech Representation Disentanglement for One-shot Voice Conversioncited by 83
  2. 2020DiscreTalk: Text-to-Speech as a Machine Translation Problemcited by 29
  3. 2024Lossless data compression by large modelscited by 22
  4. 2019Improving Opus Low Bit Rate Quality with Neural Speech Synthesiscited by 14
  5. 2021Scalable and Efficient Neural Speech Coding: A Hybrid Designcited by 7

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026TontaubeV1: Streaming Text-to-Speech with Hierarchical Codec Modeling and Bounded Context
  2. 2026StellarTTS: Sparse Temporal Embedding for Low-Latency and Robust Speech Synthesis
  3. 2025MSR-Codec: A Low-Bitrate Multi-Stream Residual Codec for High-Fidelity Speech Generation with Information Disentanglement
  4. 2026ProsoCodec: Prosody-Oriented Speech Codec for Voice Conversion
  5. 2026High-Fidelity Generative Audio Compression at 0.275kbps
  6. 2021On the Interplay Between Sparsity, Naturalness, Intelligibility, and Prosody in Speech Synthesis
  7. 2026The WER Trap: Shattering the Illusion of Unified Tokens in Speech Language Models
  8. 2026SPG-Codec: Exploring the Role and Boundaries of Semantic Priors in Ultra-Low-Bitrate Neural Speech Coding