# TS3-Codec: Transformer-Based Simple Streaming Single Codec

arXiv: https://arxiv.org/abs/2411.18803 · HTML: https://mlsanity.com/paper/2411.18803/

## Key facts

- Cited by 25 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 27 Nov 2024.

## Built on by (most cited first, 25 of 25)

- [On The Landscape of Spoken Language Models: A Comprehensive Survey](https://mlsanity.com/paper/2504.08528/) (2025)
- [Recent Advances in Discrete Speech Tokens: A Review](https://mlsanity.com/paper/2502.06490/) (2025)
- [FocalCodec: Low-Bitrate Speech Coding via Focal Modulation Networks](https://mlsanity.com/paper/2502.04465/) (2025)
- [Ming-UniAudio: Speech LLM for Joint Understanding, Generation and Editing with Unified Representation](https://mlsanity.com/paper/2511.05516/) (2025)
- [NanoCodec: Towards High-Quality Ultra Fast Speech LLM Inference](https://mlsanity.com/paper/2508.05835/) (2025)
- [Pseudo-Autoregressive Neural Codec Language Models for Efficient Zero-Shot Text-to-Speech Synthesis](https://mlsanity.com/paper/2504.10352/) (2025)
- [FocalCodec-Stream: Streaming Low-Bitrate Speech Coding via Causal Distillation](https://mlsanity.com/paper/2509.16195/) (2025)
- [MagiCodec: Simple Masked Gaussian-Injected Codec for High-Fidelity Reconstruction and Generation](https://mlsanity.com/paper/2506.00385/) (2025)
- [Unlocking Temporal Flexibility: Neural Speech Codec with Variable Frame Rate](https://mlsanity.com/paper/2505.16845/) (2025)
- [Beyond Fixed Frames: Dynamic Character-Aligned Speech Tokenization](https://mlsanity.com/paper/2601.23174/) (2026)
- [From Static Inference to Dynamic Interaction: A Survey of Streaming Large Language Models](https://mlsanity.com/paper/2603.04592/) (2026)
- [SAME: A Semantically-Aligned Music Autoencoder](https://mlsanity.com/paper/2605.18613/) (2026)
- [Finite Scalar Quantization Enables Redundant and Transmission-Robust Neural Audio Compression at Low Bit-rates](https://mlsanity.com/paper/2509.09550/) (2025)
- [Reconstruct! Don't Encode: Self-Supervised Representation Reconstruction Loss for High-Intelligibility and Low-Latency Streaming Neural Audio Codec](https://mlsanity.com/paper/2603.05887/) (2026)
- [Do Neural Codecs Generalize? A Controlled Study Across Unseen Languages and Non-Speech Tasks](https://mlsanity.com/paper/2601.12205/) (2026)
- [The Equalizer: Introducing Shape-Gain Decomposition in Neural Audio Codecs](https://mlsanity.com/paper/2602.15491/) (2026)
- [WavTTS: Towards High-Quality Zero-Shot TTS via Direct Raw Waveform Modeling](https://mlsanity.com/paper/2606.03455/) (2026)
- [SqueezeComposer: Temporal Speed-up is A Simple Trick for Long-form Music Composing](https://mlsanity.com/paper/2603.21073/) (2026)
- [Voxtral TTS](https://mlsanity.com/paper/2603.25551/) (2026)
- [Self-Guidance: Enhancing Neural Codecs via Decoder Manifold Alignment](https://mlsanity.com/paper/2606.12940/) (2026)
- [Memory Efficient Audio Synthesis with Decoupled Temporal Depth Diffusion Transformers](https://mlsanity.com/paper/2607.23811/) (2026)
- [ZipCodec: Ultra-Low-Frame-Rate Streaming Speech Coding](https://mlsanity.com/paper/2609.11642/) (2026)
- [TokenMapper: A Step Toward Interoperable Speech Token Translation](https://mlsanity.com/paper/2609.12563/) (2026)
- [MyoCodec: A Streaming Neural Codec for Electromyography](https://mlsanity.com/paper/2609.36687/) (2026)
- [GS-Codec: A Gaussian-Splatting Bottleneck for Neural Audio Coding](https://mlsanity.com/paper/2610.04651/) (2026)

## Abstract (arXiv)

Neural audio codecs (NACs) have garnered significant attention as key technologies for audio compression as well as audio representation for speech language models. While mainstream NAC models are predominantly convolution-based, the performance of NACs with a purely transformer-based, and convolution-free architecture remains unexplored. This paper introduces TS3-Codec, a Transformer-Based Simple Streaming Single Codec. TS3-Codec consists of only a stack of transformer layers with a few linear layers, offering greater simplicity and expressiveness by fully eliminating convolution layers that require careful hyperparameter tuning and large computations. Under the streaming setup, the proposed TS3-Codec achieves comparable or superior performance compared to the codec with state-of-the-art convolution-based architecture while requiring only 12% of the computation and 77% of bitrate. Furthermore, it significantly outperforms the convolution-based codec when using similar computational resources.