mlsanityData to 14 Oct 2026

Improving Joint Embedding Predictive Architecture with Diffusion Noise

arXiv 2507.15216 ↗First posted 21 Jul 2025cs.CVcited by 0

On the map: Diffusion models → Diffusion foundations and sampling

Abstract (arXiv)

Self-supervised learning has become an incredibly successful method for feature learning, widely applied to many downstream tasks. It has proven especially effective for discriminative tasks, surpassing the trending generative models. However, generative models perform better in image generation and detail enhancement. Thus, it is natural for us to find a connection between SSL and generative models to further enhance the representation capacity of SSL. As generative models can create new samples by approximating the data distribution, such modeling should also lead to a semantic understanding of the raw visual data, which is necessary for recognition tasks. This enlightens us to combine the core principle of the diffusion model: diffusion noise, with SSL to learn a competitive recognition model. Specifically, diffusion noise can be viewed as a particular state of mask that reveals a close relationship between masked image modeling (MIM) and diffusion models. In this paper, we propose N-JEPA (Noise-based JEPA) to incorporate diffusion noise into MIM by the position embedding of masked tokens. The multi-level noise schedule is a series of feature augmentations to further enhance the robustness of our model. We perform a comprehensive study to confirm its effectiveness in the classification of downstream tasks. Codes will be released soon in public.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Diffusion foundations and sampling, one of 14 directions in Diffusion models. Builds on 38 of its 44 references that are in the map. First posted on arXiv 21 Jul 2025.

Builds on

38 of 44 references.

Closest ancestors · 15

Same direction, newest first

  1. 2024Deconstructing Denoising Diffusion Models for Self-Supervised Learning25 Jan
  2. 2023Latent Consistency Models: Synthesizing High-Resolution Images with Few-Step Inference6 Oct
  3. 2023Fast Training of Diffusion Models with Masked Transformers15 Jun
  4. 2023Diffusion Models as Masked Autoencoders6 Apr
  5. 2023MDTv2: Masked Diffusion Transformer is a Strong Image Synthesizer25 Mar
  6. 2022Scalable Diffusion Models with Transformers19 Dec
  7. 2022On Distillation of Guided Diffusion Models6 Oct
  8. 2022Your ViT is Secretly a Hybrid Discriminative-Generative Diffusion Model16 Aug
  9. 2022Elucidating the Design Space of Diffusion-Based Generative Models1 Jun
  10. 2021Soft Truncation: A Universal Training Technique of Score-based Diffusion Model for High Precision Score Estimation10 Jun
  11. 2021Improved Denoising Diffusion Probabilistic Models18 Feb
  12. 2021Maximum Likelihood Training of Score-Based Diffusion Models22 Jan
3 more
  1. 2020Score-Based Generative Modeling through Stochastic Differential Equations26 Nov
  2. 2020Denoising Diffusion Implicit Models6 Oct
  3. 2020Denoising Diffusion Probabilistic Models19 Jun

Other references · 23

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,513
  3. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,403
  4. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,968
  5. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,585
  6. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,750
  7. 2019Generative Modeling by Estimating Gradients of the Data Distributioncited by 4,101
  8. 2020Unsupervised Learning of Visual Features by Contrasting Cluster Assignmentscited by 2,857
  9. 2021BEiT: BERT Pre-Training of Image Transformerscited by 2,375
  10. 2020Improved Baselines with Momentum Contrastive Learningcited by 2,287
  11. 2020Big Self-Supervised Models are Strong Semi-Supervised Learnerscited by 1,380
  12. 2021SimMIM: A Simple Framework for Masked Image Modelingcited by 1,153
11 more
  1. 2021iBOT: Image BERT Pre-Training with Online Tokenizercited by 903
  2. 2020Self-supervised Learning: Generative or Contrastivecited by 694
  3. 2021Masked Feature Prediction for Self-Supervised Visual Pre-Trainingcited by 576
  4. 2021Self-Supervised Representation Learning: Introduction, Advances and Challengescited by 148
  5. 2022What to Hide from Your Students: Attention-Guided Masked Image Modelingcited by 118
  6. 2020Towards Domain-Agnostic Contrastive Learningcited by 75
  7. 2023DreamTeacher: Pretraining Image Backbones with Deep Generative Modelscited by 37
  8. 2024Rethinking Patch Dependence for Masked Autoencoderscited by 29
  9. 2023Hallucination Improves the Performance of Unsupervised Visual Representation Learningcited by 16
  10. 2022Exploring The Role of Mean Teachers in Self-supervised Masked Auto-Encoderscited by 13
  11. 2021Training Stacked Denoising Autoencoders for Representation Learningcited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2024Denoising with a Joint-Embedding Predictive Architecture
  2. 2026DLLM-JEPA: Joint Embedding Predictive Architectures for Masked Diffusion Language Models
  3. 2025Automated Learning of Semantic Embedding Representations for Diffusion Models
  4. 2025Boosting Generative Image Modeling via Joint Image-Feature Synthesis
  5. 2026Embedding Prediction Helps Image Generation
  6. 2024How JEPA Avoids Noisy Features: The Implicit Bias of Deep Linear Self Distillation Networks
  7. 2023Masked Diffusion as Self-supervised Representation Learner
  8. 2024Simplified and Generalized Masked Diffusion for Discrete Data