mlsanityData to 14 Oct 2026

JEPA as a Neural Tokenizer: Learning Robust Speech Representations with Density Adaptive Attention

arXiv 2512.07168 ↗First posted 8 Dec 2025cs.SD cs.AI cs.LGcited by 3

Not on the map. Why?

Abstract (arXiv)

We introduce a two-stage self-supervised framework that combines the Joint-Embedding Predictive Architecture (JEPA) with a Density Adaptive Attention Mechanism (DAAM) for learning robust speech representations. Stage~1 uses JEPA with DAAM to learn semantic audio features via masked prediction in latent space, fully decoupled from waveform reconstruction. Stage~2 leverages these representations for efficient tokenization using Finite Scalar Quantization (FSQ) and a mixed-radix packing scheme, followed by high-fidelity waveform reconstruction with a HiFi-GAN decoder. By integrating Gaussian mixture-based density-adaptive gating into the JEPA encoder, the model performs adaptive temporal feature selection and discovers hierarchical speech structure at a low frame rate of 2.5~Hz. The resulting tokens (47.5 tokens/sec) provide a reversible, highly compressed, and language-model-friendly representation that is competitive with, and often more efficient than, existing neural audio codecs.

Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Builds on 15 of its 23 references that are in the map. First posted on arXiv 8 Dec 2025.

Builds on

15 of 23 references.

References · 15

  1. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,750
  2. 2020Conformer: Convolution-augmented Transformer for Speech Recognitioncited by 1,863
  3. 2021WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processingcited by 1,657
  4. 2020HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesiscited by 1,154
  5. 2023Self-Supervised Learning from Images with a Joint-Embedding Predictive Architecturecited by 1,039
  6. 2021SoundStream: An End-to-End Neural Audio Codeccited by 771
  7. 2022High Fidelity Neural Audio Compressioncited by 687
  8. 2019Libri-Light: A Benchmark for ASR with Limited or No Supervisioncited by 467
  9. 2023Finite Scalar Quantization: VQ-VAE Made Simplecited by 440
  10. 2019MelGAN: Generative Adversarial Networks for Conditional Waveform Synthesiscited by 354
  11. 2019Parallel WaveGAN: A fast waveform generation model based on generative adversarial networks with multi-resolution spectrogramcited by 340
  12. 2020Neural Networks Fail to Learn Periodic Functions and How to Fix Itcited by 100
3 more
  1. 2025DualCodec: A Low-Frame-Rate, Semantically-Enhanced Neural Audio Codec for Speech Generationcited by 21
  2. 2024Density Adaptive Attention is All You Need: Robust Parameter-Efficient Fine-Tuning Across Multiple Modalitiescited by 2
  3. 2024DAC-JAX: A JAX Implementation of the Descript Audio Codeccited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 3.

3 papers.

  1. 2026EIDOS: Latent-Space Predictive Learning for Time Series Foundation Modelscited by 1
  2. 2026ArtNet: A JEPA-Like Articulatory Predictive Framework for Robust Zero-Shot Phoneme Recognitioncited by 0
  3. 2026OLIVE: View-Augmented Latent Prediction with Waveform Reconstruction for Speech SSLcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025HH-Codec: High Compression High-fidelity Discrete Neural Codec for Spoken Language Modeling
  2. 2024JEP-KD: Joint-Embedding Predictive Architecture Based Knowledge Distillation for Visual Speech Recognition
  3. 2025Llama-Mimi: Exploring the Limits of Flattened Speech Language Modeling
  4. 2026Exploring Token-Space Manipulation in Latent Audio Tokenizers
  5. 2025Discrete Audio Tokens: More Than a Survey!
  6. 2025Explainable Disentanglement on Discrete Speech Representations for Noise-Robust ASR
  7. 2024WavTokenizer: an Efficient Acoustic Discrete Codec Tokenizer for Audio Language Modeling
  8. 2026AudioTETRIS: Token Elimination via Transitional Importance Signals for Efficient Audio LLMs