# Addressing Index Collapse of Large-Codebook Speech Tokenizer with Dual-Decoding Product-Quantized Variational Auto-Encoder

arXiv: https://arxiv.org/abs/2406.02940 · HTML: https://mlsanity.com/paper/2406.02940/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 5 Jun 2024.

## Built on by (most cited first, 3 of 3)

- [Distillation of a tractable model from the VQ-VAE](https://mlsanity.com/paper/2509.01400/) (2025)
- [Group-Wise Optimization for Self-Extensible Codebooks in Vector Quantized Models](https://mlsanity.com/paper/2510.13331/) (2025)
- [A Unified Rate-Distortion Perspective on Vector, Product, and Scalar Quantization](https://mlsanity.com/paper/2609.02107/) (2026)

## Abstract (arXiv)

VQ-VAE, as a mainstream approach of speech tokenizer, has been troubled by ``index collapse'', where only a small number of codewords are activated in large codebooks. This work proposes product-quantized (PQ) VAE with more codebooks but fewer codewords to address this problem and build large-codebook speech tokenizers. It encodes speech features into multiple VQ subspaces and composes them into codewords in a larger codebook. Besides, to utilize each VQ subspace well, we also enhance PQ-VAE via a dual-decoding training strategy with the encoding and quantized sequences. The experimental results demonstrate that PQ-VAE addresses ``index collapse" effectively, especially for larger codebooks. The model with the proposed training strategy further improves codebook perplexity and reconstruction quality, outperforming other multi-codebook VQ approaches. Finally, PQ-VAE demonstrates its effectiveness in language-model-based TTS, supporting higher-quality speech generation with larger codebooks.