Sylber 2.0: A Universal Syllable Embedding
Not on the map. Why?
Abstract (arXiv)
Scaling spoken language modeling requires speech tokens that are both efficient and universal. Recent work has proposed syllables as promising speech tokens at low temporal resolution, but existing models are constrained to English and fail to capture sufficient acoustic detail. To address this gap, we present Sylber 2.0, a self-supervised framework for coding speech at the syllable level that enables efficient temporal compression and high-fidelity reconstruction. Sylber 2.0 achieves a very low token frequency around 5 Hz, while retaining both linguistic and acoustic detail across multiple languages and expressive styles. Experiments show that it performs on par with previous models operating on high-frequency baselines. Furthermore, Sylber 2.0 enables efficient TTS modeling which can generate speech with competitive intelligibility and quality with SOTA models using only 72M parameters. Moreover, the universality of Sylber 2.0 provides more effective features for low resource ASR than previous speech coding frameworks. In sum, we establish an effective syllable-level abstraction for general spoken language.
Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Builds on 62 of its 78 references that are in the map. First posted on arXiv 29 Jan 2026.
Builds on
62 of 78 references.
References · 62
- 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
- 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,764
- 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,893
- 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 3,972
- 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,907
- 2022Flow Straight and Fast: Learning to Generate and Transfer Data with Rectified Flowcited by 3,550
- 2022A ConvNet for the 2020scited by 3,545
- 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,301
- 2020mT5: A massively multilingual pre-trained text-to-text transformercited by 1,813
- 2021WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processingcited by 1,609
- 2022High Fidelity Neural Audio Compressioncited by 673
- 2021SUPERB: Speech processing Universal PERformance Benchmarkcited by 652
38 more
- 2019LibriTTS: A Corpus Derived from LibriSpeech for Text-to-Speechcited by 639
- 2024Moshi: a speech-text foundation model for real-time dialoguecited by 536
- 2020MLS: A Large-Scale Multilingual Dataset for Speech Researchcited by 467
- 2022FLEURS: Few-shot Learning Evaluation of Universal Representations of Speechcited by 458
- 2023High-Fidelity Audio Compression with Improved RVQGANcited by 423
- 2022UTMOS: UTokyo-SaruLab System for VoiceMOS Challenge 2022cited by 381
- 2024F5-TTS: A Fairytaler that Fakes Fluent and Faithful Speech with Flow Matchingcited by 353
- 2024CosyVoice: A Scalable Multilingual Zero-shot Text-to-speech Synthesizer based on Supervised Semantic Tokenscited by 323
- 2021Layer-wise Analysis of a Self-supervised Speech Representation Modelcited by 293
- 2024Seed-TTS: A Family of High-Quality Versatile Speech Generation Modelscited by 293
- 2024CosyVoice 2: Scalable Streaming Speech Synthesis with Large Language Modelscited by 282
- 2022Self-Supervised Speech Representation Learning: A Reviewcited by 258
- 2024NaturalSpeech 3: Zero-Shot Speech Synthesis with Factorized Codec and Diffusion Modelscited by 222
- 2024Emilia: An Extensive, Multilingual, and Diverse Speech Dataset for Large-Scale Speech Generationcited by 200
- 2024MaskGCT: Zero-Shot Text-to-Speech with Masked Generative Codec Transformercited by 159
- 2022Comparative layer-wise analysis of self-supervised speech modelscited by 155
- 2023Vocos: Closing the gap between time-domain and Fourier-based neural vocoders for high-quality audio synthesiscited by 150
- 2023UniMax: Fairer and more Effective Language Sampling for Large-Scale Multilingual Pretrainingcited by 147
- 2024WavTokenizer: an Efficient Acoustic Discrete Codec Tokenizer for Audio Language Modelingcited by 124
- 2020AISHELL-3: A Multi-speaker Mandarin TTS Corpus and the Baselinescited by 111
- 2018CREPE: A Convolutional Representation for Pitch Estimationcited by 109
- 2025Spark-TTS: An Efficient LLM-Based Text-to-Speech Model with Single-Stream Decoupled Speech Tokenscited by 109
- 2023EXPRESSO: A Benchmark and Analysis of Discrete Expressive Speech Resynthesiscited by 93
- 2023FunASR: A Fundamental End-to-End Speech Recognition Toolkitcited by 92
- 2022ContentVec: An Improved Self-Supervised Speech Representation by Disentangling Speakerscited by 73
- 2024Self-Supervised Speech Representations are More Phonetic than Semanticcited by 70
- 2024FireRedTTS: A Foundation Text-To-Speech Framework for Industry-Level Generative Speech Applicationscited by 68
- 2024mHuBERT-147: A Compact Multilingual HuBERT Modelcited by 64
- 2025DiTAR: Diffusion Transformer Autoregressive Modeling for Speech Generationcited by 56
- 2024Scaling Transformers for Low-Bitrate High-Quality Speech Codingcited by 48
- 2023Analysing Discrete Self Supervised Speech Representation for Spoken Language Modelingcited by 47
- 2025Recent Advances in Discrete Speech Tokens: A Reviewcited by 42
- 2024Autoregressive Diffusion Transformer for Text-to-Speech Synthesiscited by 34
- 2022Evidence of Vocal Tract Articulation in Self-Supervised Learning of Speechcited by 30
- 2024GLOBE: A High-quality English Corpus with Global Accents for Zero-shot Speaker Adaptive Text-to-Speechcited by 30
- 2024GTSinger: A Global Multi-Technique Singing Corpus with Realistic Music Scores for All Singing Taskscited by 29
- 2020ICASSP 2021 Deep Noise Suppression Challengecited by 27
- 2024SyllableLM: Learning Coarse Semantic Units for Speech Language Modelscited by 24
50 most cited of 62 All 62 (JSON)
Built on by
Citing papers per year: 2026: 2.
2 papers.