mlsanityData to 7 Oct 2026

Echo: A Joint-Embedding Predictive Architecture for Speaker Diarization and Speech Recognition in a Shared Latent Space

arXiv 2606.01909 ↗First posted 1 Jun 2026cs.SD cs.AI eess.AScited by 0

Not on the map. Why?

Abstract (arXiv)

We present Echo, a proof-of-concept audio system built around a single 25 M-parameter ViT encoder. The encoder is pretrained with a JEPA objective and then specialised by stages to carry speaker identity, phonetic content, and dynamic source routing in the same 512-dimensional latent space, with no per-task fine-tuning at deployment. Light heads handle diarization (ArcFace + VBx) and dynamic source separation (null-target K-set prediction). On synthetic VoxCeleb2 mixtures with unknown K, the canonical stack reaches 15.00% blind DER, 97.80% PIT separation accuracy with +9.52 dB latent SI-SDR, and a +53.50-point speaker/content factorisation gap on a held-out k-NN probe. The point of Echo is not a new SOTA on any single task but the joint coexistence of three tasks on one encoder at this footprint. We document the design stage by stage, report the dead-ends, and identify the structural wall on end-to-end ASR through the VQ bottleneck that still bounds the PoC.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 14 of its 17 references that are in the map. First posted on arXiv 1 Jun 2026.

Builds on

14 of 17 references.

References · 14

  1. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,853
  2. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,301
  3. 2021WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processingcited by 1,609
  4. 2023Self-Supervised Learning from Images with a Joint-Embedding Predictive Architecturecited by 982
  5. 2018VoxCeleb2: Deep Speaker Recognitioncited by 862
  6. 2022data2vec: A General Framework for Self-supervised Learning in Speech, Vision and Languagecited by 718
  7. 2016Permutation Invariant Training of Deep Models for Speaker-Independent Multi-talker Speech Separationcited by 189
  8. 2020End-to-End Speaker Diarization for an Unknown Number of Speakers with Encoder-Decoder Based Attractorscited by 70
  9. 2020Bayesian HMM clustering of x-vector sequences (VBx) in speaker diarization: theory, implementation and analysis on standard taskscited by 69
  10. 2024Masked Modeling Duo: Towards a Universal Audio Pre-training Frameworkcited by 23
  11. 2025Audio-JEPA: Joint-Embedding Predictive Architecture for Audio Representation Learningcited by 17
  12. 2023TS-SEP: Joint Diarization and Separation Conditioned on Estimated Speaker Embeddingscited by 12
2 more
  1. 2022EEND-SS: Joint End-to-End Neural Speaker Diarization and Speech Separation for Flexible Number of Speakerscited by 10
  2. 2024PixIT: Joint Training of Speaker Diarization and Speech Separation from Real-world Multi-speaker Recordingscited by 8

Built on by

Not cited yet.