mlsanityData to 14 Oct 2026

Speech Codec Probing from Semantic and Phonetic Perspectives

arXiv 2603.10371 ↗First posted 11 Mar 2026eess.AS cs.CLcited by 1

Not on the map. Why?

Abstract (arXiv)

Speech tokenizers are essential for connecting speech to large language models (LLMs) in multimodal systems. Speech tokenizers are expected to preserve both semantic and acoustic information for downstream understanding and generation tasks. However, emerging evidence suggests that the term "semantic" in speech processing does not align with linguistic lexical-semantic, leading to a mismatch between speech and text modality. In this paper, we systematically analyze the information encoded by several widely used speech tokenizers, evaluating their lexical-semantic and phonetic content through three tasks. Our results show that current tokenizers primarily capture phonetic rather than lexical-semantic structure, deriving practical implications for the design of next-generation speech tokenization methods. Code is released to public at https://github.com/Alexuan/codec_probing_release.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 24 of its 35 references that are in the map. First posted on arXiv 11 Mar 2026.

Builds on

24 of 35 references.

References · 24

  1. 2024GPT-4o System Cardcited by 4,588
  2. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,342
  3. 2019Similarity of Neural Network Representations Revisitedcited by 1,755
  4. 2021WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processingcited by 1,657
  5. 2021SoundStream: An End-to-End Neural Audio Codeccited by 771
  6. 2025Qwen2.5-Omni Technical Reportcited by 736
  7. 2022High Fidelity Neural Audio Compressioncited by 687
  8. 2021Vector-quantized Image Modeling with Improved VQGANcited by 658
  9. 2022AudioLM: a Language Modeling Approach to Audio Generationcited by 616
  10. 2024Moshi: a speech-text foundation model for real-time dialoguecited by 568
  11. 2023High-Fidelity Audio Compression with Improved RVQGANcited by 431
  12. 2024CosyVoice: A Scalable Multilingual Zero-shot Text-to-speech Synthesizer based on Supervised Semantic Tokenscited by 334
12 more
  1. 2018Insights on representational similarity in neural networks with canonical correlationcited by 324
  2. 2020Neural Networks Fail to Learn Periodic Functions and How to Fix Itcited by 100
  3. 2024Self-Supervised Speech Representations are More Phonetic than Semanticcited by 71
  4. 2025MiMo-Audio: Audio Language Models are Few-Shot Learnerscited by 65
  5. 2020SEANet: A Multi-modal Speech Enhancement Networkcited by 37
  6. 2025XY-Tokenizer: Mitigating the Semantic-Acoustic Conflict in Low-Bitrate Speech Codecscited by 23
  7. 2021A multispeaker dataset of raw and reconstructed speech production real-time MRI video and 3D volumetric imagescited by 21
  8. 2025Closing the Gap Between Text and Speech Understanding in LLMscited by 17
  9. 2025Understanding the Modality Gap: An Empirical Study on the Speech-Text Alignment Mechanism of Large Speech Language Modelscited by 13
  10. 2024HAM-TTS: Hierarchical Acoustic Modeling for Token-Based Zero-Shot Text-to-Speech with Model and Data Scalingcited by 6
  11. 2024SonicSim: A customizable simulation platform for speech processing in moving sound source scenarioscited by 4
  12. 2023Improving Generalization Ability of Countermeasures for New Mismatch Scenario by Combining Multiple Advanced Regularization Termscited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026DriftSE: Speech Enhancement with Generative Driftingcited by 0