mlsanityData to 14 Oct 2026

Segmentation-Variant Codebooks for Preservation of Paralinguistic and Prosodic Information

arXiv 2505.15667 ↗First posted 21 May 2025eess.AS cs.CL cs.SDcited by 4

Not on the map. Why?

Abstract (arXiv)

Quantization in SSL speech models (e.g., HuBERT) improves compression and performance in tasks like language modeling, resynthesis, and text-to-speech but often discards prosodic and paralinguistic information (e.g., emotion, prominence). While increasing codebook size mitigates some loss, it inefficiently raises bitrates. We propose Segmentation-Variant Codebooks (SVCs), which quantize speech at distinct linguistic units (frame, phone, word, utterance), factorizing it into multiple streams of segment-specific discrete features. Our results show that SVCs are significantly more effective at preserving prosodic and paralinguistic information across probing tasks. Additionally, we find that pooling before rather than after discretization better retains segment-level information. Resynthesis experiments further confirm improved style realization and slightly improved quality while preserving intelligibility.

Key facts. Cited by 4 papers in the map (data through 14 Oct 2026). Builds on 26 of its 33 references that are in the map. First posted on arXiv 21 May 2025.

Builds on

26 of 33 references.

References · 26

  1. 2017Neural Discrete Representation Learningcited by 4,712
  2. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,905
  3. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,342
  4. 2020HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesiscited by 1,154
  5. 2021SoundStream: An End-to-End Neural Audio Codeccited by 771
  6. 2022High Fidelity Neural Audio Compressioncited by 687
  7. 2021SUPERB: Speech processing Universal PERformance Benchmarkcited by 666
  8. 2022UTMOS: UTokyo-SaruLab System for VoiceMOS Challenge 2022cited by 390
  9. 2019vq-wav2vec: Self-Supervised Learning of Discrete Speech Representationscited by 340
  10. 2023EXPRESSO: A Benchmark and Analysis of Discrete Expressive Speech Resynthesiscited by 98
  11. 2021A Fine-tuned Wav2vec 2.0/HuBERT Benchmark For Speech Emotion Recognition, Speaker Verification and Spoken Language Understandingcited by 82
  12. 2021Exploring Wav2vec 2.0 fine-tuning for improved speech emotion recognitioncited by 49
14 more
  1. 2023Analysing Discrete Self Supervised Speech Representation for Spoken Language Modelingcited by 47
  2. 2023Exploring Speech Recognition, Translation, and Understanding with Discrete Speech Units: A Comparative Studycited by 42
  3. 2022Exploration of A Self-Supervised Speech Model: A Study on Emotional Corporacited by 31
  4. 2023EmphAssess : a Prosodic Benchmark on Assessing Emphasis Transfer in Speech-to-Speech Modelscited by 22
  5. 2022Analyzing Acoustic Word Embeddings from Pre-trained Self-supervised Speech Modelscited by 21
  6. 2024A layer-wise analysis of Mandarin and English suprasegmentals in SSL speech modelscited by 18
  7. 2023Speech Emotion Diarization: Which Emotion Appears When?cited by 13
  8. 2024The Interspeech 2024 Challenge on Speech Processing Using Discrete Unitscited by 12
  9. 2024DiscreteSLU: A Large Language Model with Self-Supervised Discrete Speech Units for Spoken Language Understandingcited by 10
  10. 2024Crossmodal ASR Error Correction with Discrete Speech Unitscited by 9
  11. 2024Layer-Wise Analysis of Self-Supervised Acoustic Word Embeddings: A Study on Speech Emotion Recognitioncited by 8
  12. 2024EMO-Codec: An In-Depth Look at Emotion Preservation capacity of Legacy and Neural Codec Models With Subjective and Objective Evaluationscited by 8
  13. 2023On the Use of Self-Supervised Speech Representations in Spontaneous Speech Synthesiscited by 4
  14. 2024Text-to-Speech for Unseen Speakers via Low-Complexity Discrete Unit-Based Frame Selectioncited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 4.

4 papers.

  1. 2026F-Actor: Controllable Conversational Behaviour in Full-Duplex Modelscited by 3
  2. 2026Lexical Tone is Hard to Quantize: Probing Discrete Speech Units in Mandarin and Yor`ub'acited by 1
  3. 2026Recovering Performance in Speech Emotion Recognition from Discrete Tokens via Multi-Layer Fusion and Paralinguistic Feature Integrationcited by 0
  4. 2026On the Effect of Segmentation Width and Cluster Size on Speech Resynthesis and Continuation in Generative Spoken Language Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Disentangling segmental and prosodic factors to non-native speech comprehensibility
  2. 2026ASSET: Adaptive Speech Segmentation via Reinforcement Learning
  3. 2026Text Scores Do Not Establish Performance on Lexically Non-Diagnostic Speech Tasks: A Qwen2-Audio Quantization Case Study
  4. 2025Speech Tokenizer is Key to Consistent Representation
  5. 2026LACE: Layer-Wise Compression for Dynamic Frame Rate Codecs
  6. 2021On the Interplay Between Sparsity, Naturalness, Intelligibility, and Prosody in Speech Synthesis
  7. 2026Language-Statistical Analysis of Neural Audio Codec Tokens Across Architectures, Corpora, and Noise Conditions
  8. 2025Segmentwise Pruning in Audio-Language Models