mlsanityData to 19 Oct 2026

SpecTokenizer: A Lightweight Streaming Codec in the Compressed Spectrum Domain

arXiv 2510.21209 ↗First posted 24 Oct 2025eess.AS cs.SDcited by 2

On the map: Speech and audio language models → Vocoders and speech synthesis

Abstract (arXiv)

Neural Audio Codecs (NACs) have gained growing attention in recent years as technologies for audio compression and audio representation in speech language models. While mainstream NACs typically require G-level computation and M-level parameters, the performance of lightweight and streaming NACs remains underexplored. This paper proposes SpecTokenizer, a lightweight streaming codec that operates in the compressed spectral domain. Composed solely of alternating CNN and RNN layers, SpecTokenizer achieves greater efficiency and better representational capability through multi-scale modeling in the compressed spectrum domain. At 4 kbps, the proposed SpecTokenizer achieves comparable or superior performance compared to the codec with state-of-the-art lightweight architecture while requiring only 20% of the computation and 10% of the parameters. Furthermore, it significantly outperforms the codec when using similar computational and storage resources.

Key facts. Cited by 2 papers in the map (data through 19 Oct 2026). Placed in Vocoders and speech synthesis, one of 11 directions in Speech and audio language models. Builds on 32 of its 35 references that are in the map. First posted on arXiv 24 Oct 2025.

Builds on

32 of 35 references.

Closest ancestors · 14

Same direction, newest first

  1. 2024CosyVoice 2: Scalable Streaming Speech Synthesis with Large Language Models13 Dec
  2. 2024TS3-Codec: Transformer-Based Simple Streaming Single Codec27 Nov
  3. 2024A Comparative Study of Discrete Speech Tokens for Semantic-Related Tasks with Large Language Models13 Nov
  4. 2024MDCTCodec: A Lightweight MDCT-based Neural Audio Codec towards High Sampling Rate and Low Bitrate Scenarios1 Nov
  5. 2024ESPnet-Codec: Comprehensive Training and Evaluation of Neural Codecs for Audio, Music, and Speech24 Sep
  6. 2024Codec-SUPERB @ SLT 2024: A lightweight benchmark for neural audio codec models21 Sep
  7. 2024SoCodec: A Semantic-Ordered Multi-Stream Speech Codec for Efficient Language Model Based Text-to-Speech Synthesis2 Sep
  8. 2024WavTokenizer: an Efficient Acoustic Discrete Codec Tokenizer for Audio Language Modeling29 Aug
  9. 2024ESC: Efficient Speech Coding with Cross-Scale Residual Vector Quantized Transformers30 Apr
  10. 2024Towards audio language modeling -- an overview20 Feb
  11. 2023FunCodec: A Fundamental, Reproducible and Integrable Open-source Toolkit for Neural Speech Codec14 Sep
  12. 2023HiFi-Codec: Group-residual Vector quantization for High Fidelity Audio Codec4 May
2 more
  1. 2022End-to-End Neural Speech Coding for Real-Time Communications24 Jan
  2. 2021SoundStream: An End-to-End Neural Audio Codec7 Jul

Other references · 18

  1. 2017Neural Discrete Representation Learningcited by 4,843
  2. 2020HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesiscited by 1,462
  3. 2022High Fidelity Neural Audio Compressioncited by 917
  4. 2019LibriTTS: A Corpus Derived from LibriSpeech for Text-to-Speechcited by 864
  5. 2023High-Fidelity Audio Compression with Improved RVQGANcited by 619
  6. 2020Jukebox: A Generative Model for Musiccited by 564
  7. 2022UTMOS: UTokyo-SaruLab System for VoiceMOS Challenge 2022cited by 552
  8. 2023AudioGPT: Understanding and Generating Speech, Music, Sound, and Talking Headcited by 288
  9. 2023UniAudio: An Audio Foundation Model Toward Universal Audio Generationcited by 162
  10. 2020Neural Networks Fail to Learn Periodic Functions and How to Fix Itcited by 121
  11. 2025Recent Advances in Discrete Speech Tokens: A Reviewcited by 65
  12. 2024Codec-SUPERB: An In-Depth Analysis of Sound Codec Modelscited by 52
6 more
  1. 2023Music Source Separation with Band-Split RoPE Transformercited by 43
  2. 2024UniAudio 1.5: Large Language Model-driven Audio Codec is A Few-shot Audio Task Learnercited by 41
  3. 2025GenSE: Generative Speech Enhancement via Language Models using Hierarchical Modelingcited by 36
  4. 2021Generative Speech Coding with Predictive Variance Regularizationcited by 30
  5. 2024ICASSP 2024 Speech Signal Improvement Challengecited by 30
  6. 2023Analysis of XLS-R for Speech Quality Assessmentcited by 4

Built on by

Cited by year (2026 to 19 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026Benchmarking Neural Speech Compression from a Rate-Distortion Perspectivecited by 1
  2. 2026ReLMCodec: Designing Predictable Speech Tokens from Pre-Quantization Phoneme Structurecited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026ZipCodec: Ultra-Low-Frame-Rate Streaming Speech Coding
  2. 2024HILCodec: High-Fidelity and Lightweight Neural Audio Codec
  3. 2025ALMTokenizer: A Low-bitrate and Semantic-rich Audio Codec Tokenizer for Audio Language Modeling
  4. 2025FocalCodec-Stream: Streaming Low-Bitrate Speech Coding via Causal Distillation
  5. 2024Towards Audio Codec-based Speech Separation
  6. 2024A Closer Look at Neural Codec Resynthesis: Bridging the Gap between Codec and Waveform Generation
  7. 2026GS-Codec: A Gaussian-Splatting Bottleneck for Neural Audio Coding
  8. 2025SpectroStream: A Versatile Neural Codec for General Audio