mlsanityData to 14 Oct 2026

A-JEPA: Joint-Embedding Predictive Architecture Can Listen

arXiv 2311.15830 ↗First posted 27 Nov 2023cs.SD cs.CV eess.AScited by 53

Not on the map. Why?

Abstract (arXiv)

This paper presents that the masked-modeling principle driving the success of large foundational vision models can be effectively applied to audio by making predictions in a latent space. We introduce Audio-based Joint-Embedding Predictive Architecture (A-JEPA), a simple extension method for self-supervised learning from the audio spectrum. Following the design of I-JEPA, our A-JEPA encodes visible audio spectrogram patches with a curriculum masking strategy via context encoder, and predicts the representations of regions sampled at well-designed locations. The target representations of those regions are extracted by the exponential moving average of context encoder, i.e., target encoder, on the whole spectrogram. We find it beneficial to transfer random block masking into time-frequency aware masking in a curriculum manner, considering the complexity of highly correlated in local time and frequency in audio spectrograms. To enhance contextual semantic understanding and robustness, we fine-tune the encoder with a regularized masking on target datasets, instead of input dropping or zero. Empirically, when built with Vision Transformers structure, we find A-JEPA to be highly scalable and sets new state-of-the-art performance on multiple audio and speech classification tasks, outperforming other recent models that use externally supervised pre-training.

Key facts. Cited by 53 papers in the map (data through 14 Oct 2026). Builds on 54 of its 73 references that are in the map. First posted on arXiv 27 Nov 2023.

Builds on

54 of 73 references.

References · 54

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2020Language Models are Few-Shot Learnerscited by 32,419
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  5. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  6. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,139
  7. 2018Representation Learning with Contrastive Predictive Codingcited by 7,777
  8. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,403
  9. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,905
  10. 2020Unsupervised Learning of Visual Features by Contrasting Cluster Assignmentscited by 2,857
  11. 2021BEiT: BERT Pre-Training of Image Transformerscited by 2,375
  12. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,342
38 more
  1. 2016Context Encoders: Feature Learning by Inpaintingcited by 1,428
  2. 2019SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognitioncited by 1,380
  3. 2022VideoMAE: Masked Autoencoders are Data-Efficient Learners for Self-Supervised Video Pre-Trainingcited by 1,322
  4. 2023Self-Supervised Learning from Images with a Joint-Embedding Predictive Architecturecited by 1,039
  5. 2021iBOT: Image BERT Pre-Training with Online Tokenizercited by 903
  6. 2018Speech Commands: A Dataset for Limited-Vocabulary Speech Recognitioncited by 752
  7. 2022data2vec: A General Framework for Self-supervised Learning in Speech, Vision and Languagecited by 727
  8. 2016CNN Architectures for Large-Scale Audio Classificationcited by 704
  9. 2019Data-Efficient Image Recognition with Contrastive Predictive Codingcited by 673
  10. 2021SUPERB: Speech processing Universal PERformance Benchmarkcited by 666
  11. 2021AST: Audio Spectrogram Transformercited by 643
  12. 2021Masked Feature Prediction for Self-Supervised Visual Pre-Trainingcited by 576
  13. 2019PANNs: Large-Scale Pretrained Audio Neural Networks for Audio Pattern Recognitioncited by 574
  14. 2022Masked Autoencoders As Spatiotemporal Learnerscited by 429
  15. 2021Attention Bottlenecks for Multimodal Fusioncited by 361
  16. 2019On The Power of Curriculum Learning in Training Deep Networkscited by 286
  17. 2017Objects that Soundcited by 256
  18. 2021Efficient Training of Audio Transformers with Patchoutcited by 226
  19. 2019Competence-based Curriculum Learning for Neural Machine Translationcited by 202
  20. 2019Mockingjay: Unsupervised Speech Representation Learning with Deep Bidirectional Transformer Encoderscited by 198
  21. 2022HTS-AT: A Hierarchical Token-Semantic Audio Transformer for Sound Classification and Detectioncited by 176
  22. 2021SSAST: Self-Supervised Audio Spectrogram Transformercited by 169
  23. 2020GridMask Data Augmentationcited by 123
  24. 2022VICRegL: Self-Supervised Learning of Local Visual Featurescited by 118
  25. 2022Efficient Self-supervised Learning with Contextualized Target Representations for Vision, Speech and Languagecited by 110
  26. 2021The Benefit Of Temporally-Strong Labels In Audio Event Classificationcited by 83
  27. 2021PSLA: Improving Audio Tagging with Pretraining, Sampling, Labeling, and Aggregationcited by 82
  28. 2022MAE-AST: Masked Autoencoding Audio Spectrogram Transformercited by 82
  29. 2022Robust Self-Supervised Audio-Visual Speech Recognitioncited by 75
  30. 2020Learning Representations by Predicting Bags of Visual Wordscited by 73
  31. 2021High Fidelity Visualization of What Your Self-Supervised Representation Knows Aboutcited by 66
  32. 2022Masked Spectrogram Modeling using Masked Autoencoders for Learning General-purpose Audio Representationcited by 52
  33. 2022Masked Spectrogram Prediction For Self-Supervised Audio Pre-Trainingcited by 51
  34. 2021Space-Time Crop & Attend: Improving Cross-modal Video Representation Learningcited by 31
  35. 2021ERANNs: Efficient Residual Audio Neural Networks for Audio Pattern Recognitioncited by 29
  36. 2021Conformer-Based Self-Supervised Learning for Non-Speech Audio Taskscited by 22
  37. 2019Fast Image Caption Generation with Position Alignmentcited by 21
  38. 2018Image Captioningcited by 14

50 most cited of 54 All 54 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 1; 2024: 10; 2025: 15; 2026: 27.

53 papers.

Sort:

By area:

  1. 2024Aligning Cyber Space with Physical World: A Comprehensive Survey on Embodied AIcited by 179
  2. 2024Is Sora a World Simulator? A Comprehensive Survey on General World Models and Beyondcited by 111
  3. 2024World Models for Autonomous Driving: An Initial Surveycited by 80
  4. 2025SkyReels-A2: Compose Anything in Video Diffusion Transformerscited by 73
  5. 2025VL-JEPA: Joint Embedding Predictive Architecture for Vision-languagecited by 68
  6. 2026World Action Models: The Next Frontier in Embodied AIcited by 50
  7. 2024Foundation Models for Music: A Surveycited by 37
  8. 2025Embodied AI: From LLMs to World Modelscited by 24
  9. 2024FLUX that Plays Musiccited by 18
  10. 2025Audio-JEPA: Joint-Embedding Predictive Architecture for Audio Representation Learningcited by 17
  11. 2025SSLAM: Enhancing Self-Supervised Models with Audio Mixtures for Polyphonic Soundscapescited by 16
  12. 2025Ingredients: Blending Custom Photos with Video Diffusion Transformerscited by 13
38 more
  1. 2024Video Diffusion Transformers are In-Context Learnerscited by 12
  2. 2026World Action Models: A Surveycited by 12
  3. 2025Recurrent Video Masked Autoencoderscited by 10
  4. 2024Time-Series JEPA for Predictive Remote Control under Capacity-Limited Networkscited by 9
  5. 2024LaT-PFN: A Joint Embedding Predictive Architecture for In-context Time-series Forecastingcited by 8
  6. 2025ACT-JEPA: Novel Joint-Embedding Predictive Architecture for Efficient Policy Representation Learningcited by 8
  7. 2024Music Consistency Modelscited by 7
  8. 2023Prospective Role of Foundation Models in Advancing Autonomous Vehiclescited by 7
  9. 2024Learning Latent Wireless Dynamics from Channel State Informationcited by 7
  10. 2025CrossJEPA: Cross-Modal Joint-Embedding Predictive Architecture for Efficient 3D Representation Learning from 2D Imagescited by 4
  11. 2026AeroJEPA: Learning Semantic Latent Representations for Scalable 3D Aerodynamic Field Modelingcited by 4
  12. 2026Self-Distillation of Hidden Layers for Self-Supervised Representation Learningcited by 3
  13. 2025Discrete JEPA: Learning Discrete Token Representations without Reconstructioncited by 2
  14. 2026Pantagruel: Unified Self-Supervised Encoders for French Text and Speechcited by 2
  15. 2026From Alignment to Prediction: A Study of Self-Supervised Learning and Predictive Representation Learningcited by 2
  16. 2026MJEPA: A Simple and Scalable Joint-Embedding Predictive Architecture for Audio-Visual Learningcited by 2
  17. 2026Music-JEPA: Learning a World Model of Sound from Actioncited by 2
  18. 2025Chirp Localization via Fine-Tuned Transformer Model: A Proof-of-Concept Studycited by 1
  19. 2026Hierarchical Molecular Representation Learning via Fragment-Based Self-Supervised Embedding Predictioncited by 1
  20. 2026AudioMosaic: Contrastive Masked Audio Representation Learningcited by 1
  21. 2026Joint-Embedding Predictive Architecture for Sensor-based Activity Recognitioncited by 1
  22. 2026Scalable and Efficient Joint Spiking Embedding Predictive Architecture for Large-Scale Dynamic Graphscited by 1
  23. 2025Predict, Cluster, Refine: A Joint Embedding Predictive Self-Supervised Framework for Graph Representation Learningcited by 0
  24. 2025Towards Unified Approaches in Self-Supervised Event Stream Modeling: Progress and Prospectscited by 0
  25. 2025BiRQ: Bi-Level Self-Labeling Random Quantization for Self-Supervised Speech Recognitioncited by 0
  26. 2025Unsupervised Transformer Pre-Training for Images: Self-Distillation, Mean Teachers, and Random Cropscited by 0
  27. 2026Representation-Regularized Convolutional Audio Transformer for Audio Understandingcited by 0
  28. 2026Stylistic-STORM (ST-STORM) : Perceiving the Semantic Nature of Appearancecited by 0
  29. 2026PairAlign: A Framework for Autoregressive Tokenization via Self-Alignment with Applications to Audio Tokenizationcited by 0
  30. 2026BEST-RQ-2: Contextualize-Then-Predict, a Two-Step Approach for Self-Supervised Audio Representationscited by 0
  31. 2026AGE: Adaptive-masking for Graph Embedding in Graph Retrieval-Augmented Generationcited by 0
  32. 2026Joint-Embedding Predictive Architecture for Solar PV Panel Fault Classificationcited by 0
  33. 2026Bar-JEPA: Extracting Values from Bar Chart with Joint-Embedding Predictive Architecturecited by 0
  34. 2026Listening Forward: Next Patch Embedding Prediction Enables Scalable Audio Learnerscited by 0
  35. 2026Distributed JEPA: A Self-Supervised Framework for Energy Forecastingcited by 0
  36. 2026Mask-Aware Execution for Efficient JEPA Trainingcited by 0
  37. 2026I Act Therefore I Am: When Is JEPA's Action-Conditioning Enough to Learn Causal Mechanisms?cited by 0
  38. 2026JEPA Learns What the Mask Leaves Unrecoverablecited by 0

50 most cited of 53 All 53 (JSON)