# Impact of Frame Rates on Speech Tokenizer: A Case Study on Mandarin and English

arXiv: https://arxiv.org/abs/2505.17076 · HTML: https://mlsanity.com/paper/2505.17076/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 19 of its 27 references that are in the map.
- First posted on arXiv 20 May 2025.

## Builds on (most cited first, 19 of 19)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Neural Discrete Representation Learning](https://mlsanity.com/paper/1711.00937/) (2017)
- [Robust Speech Recognition via Large-Scale Weak Supervision](https://mlsanity.com/paper/2212.04356/) (2022)
- [SoundStream: An End-to-End Neural Audio Codec](https://mlsanity.com/paper/2107.03312/) (2021)
- [Neural Codec Language Models are Zero-Shot Text to Speech Synthesizers](https://mlsanity.com/paper/2301.02111/) (2023)
- [High Fidelity Neural Audio Compression](https://mlsanity.com/paper/2210.13438/) (2022)
- [Qwen-Audio: Advancing Universal Audio Understanding via Unified Large-Scale Audio-Language Models](https://mlsanity.com/paper/2311.07919/) (2023)
- [Moshi: a speech-text foundation model for real-time dialogue](https://mlsanity.com/paper/2410.00037/) (2024)
- [BEATs: Audio Pre-Training with Acoustic Tokenizers](https://mlsanity.com/paper/2212.09058/) (2022)
- [CosyVoice: A Scalable Multilingual Zero-shot Text-to-speech Synthesizer based on Supervised Semantic Tokens](https://mlsanity.com/paper/2407.05407/) (2024)
- [GLM-4-Voice: Towards Intelligent and Human-Like End-to-End Spoken Chatbot](https://mlsanity.com/paper/2412.02612/) (2024)
- [WavTokenizer: an Efficient Acoustic Discrete Codec Tokenizer for Audio Language Modeling](https://mlsanity.com/paper/2408.16532/) (2024)
- [AISHELL-2: Transforming Mandarin ASR Research Into Industrial Scale](https://mlsanity.com/paper/1808.10583/) (2018)
- [SpeechTokenizer: Unified Speech Tokenizer for Speech Large Language Models](https://mlsanity.com/paper/2308.16692/) (2023)
- [Step-Audio: Unified Understanding and Generation in Intelligent Speech Interaction](https://mlsanity.com/paper/2502.11946/) (2025)
- [GenSE: Generative Speech Enhancement via Language Models using Hierarchical Modeling](https://mlsanity.com/paper/2502.02942/) (2025)
- [Aligning Speech to Languages to Enhance Code-switching Speech Recognition](https://mlsanity.com/paper/2403.05887/) (2024)
- [PHO-LID: A Unified Model Incorporating Acoustic-Phonetic and Phonotactic Information for Language Identification](https://mlsanity.com/paper/2203.12366/) (2022)
- [FaceSpeak: Expressive and High-Quality Speech Synthesis from Human Portraits of Different Styles](https://mlsanity.com/paper/2501.03181/) (2025)

## Abstract (arXiv)

The speech tokenizer plays a crucial role in recent speech tasks, generally serving as a bridge between speech signals and language models. While low-frame-rate codecs are widely employed as speech tokenizers, the impact of frame rates on speech tokens remains underexplored. In this study, we investigate how varying frame rates affect speech tokenization by examining Mandarin and English, two typologically distinct languages. We encode speech at different frame rates and evaluate the resulting semantic tokens in the speech recognition task. Our findings reveal that frame rate variations influence speech tokenization differently for each language, highlighting the interplay between frame rates, phonetic density, and language-specific acoustic features. The results provide insights into optimizing frame rate selection for speech tokenizers, with implications for automatic speech recognition, text-to-speech, and other speech-related applications.