mlsanityData to 13 Oct 2026

Denoising with a Joint-Embedding Predictive Architecture

arXiv 2410.03755 ↗First posted 2 Oct 2024cs.LG cs.CVcited by 13

On the map: Diffusion models → Diffusion foundations and sampling

Abstract (arXiv)

Joint-embedding predictive architectures (JEPAs) have shown substantial promise in self-supervised representation learning, yet their application in generative modeling remains underexplored. Conversely, diffusion models have demonstrated significant efficacy in modeling arbitrary probability distributions. In this paper, we introduce Denoising with a Joint-Embedding Predictive Architecture (D-JEPA), pioneering the integration of JEPA within generative modeling. By recognizing JEPA as a form of masked image modeling, we reinterpret it as a generalized next-token prediction strategy, facilitating data generation in an auto-regressive manner. Furthermore, we incorporate diffusion loss to model the per-token probability distribution, enabling data generation in a continuous space. We also adapt flow matching loss as an alternative to diffusion loss, thereby enhancing the flexibility of D-JEPA. Empirically, with increased GFLOPs, D-JEPA consistently achieves lower FID scores with fewer training epochs, indicating its good scalability. Our base, large, and huge models outperform all previous generative models across all scales on ImageNet conditional generation benchmarks. Beyond image generation, D-JEPA is well-suited for other continuous data modeling, including video and audio.

Key facts. Cited by 13 papers in the map (data through 13 Oct 2026). Placed in Diffusion foundations and sampling, one of 14 directions in Diffusion models. Builds on 109 of its 125 references that are in the map. First posted on arXiv 2 Oct 2024.

Builds on

109 of 125 references.

Closest ancestors · 16

Same direction, newest first

  1. 2024Lumina-Next: Making Lumina-T2X Stronger and Faster with Next-DiT5 Jun
  2. 2024Scaling Rectified Flow Transformers for High-Resolution Image Synthesis5 Mar
  3. 2024SiT: Exploring Flow and Diffusion-based Generative Models with Scalable Interpolant Transformers16 Jan
  4. 2023DiffiT: Diffusion Vision Transformers for Image Generation4 Dec
  5. 2023PixArt-α: Fast Training of Diffusion Transformer for Photorealistic Text-to-Image Synthesis30 Sep
  6. 2023Diffusion Models as Masked Autoencoders6 Apr
  7. 2023Understanding Diffusion Objectives as the ELBO with Simple Data Augmentation1 Mar
  8. 2022Scalable Diffusion Models with Transformers19 Dec
  9. 2022Diffusion Models in Vision: A Survey10 Sep
  10. 2022Elucidating the Design Space of Diffusion-Based Generative Models1 Jun
  11. 2021High-Resolution Image Synthesis with Latent Diffusion Models20 Dec
  12. 2021Diffusion Models Beat GANs on Image Synthesis11 May
4 more
  1. 2021Improved Denoising Diffusion Probabilistic Models18 Feb
  2. 2020Score-Based Generative Modeling through Stochastic Differential Equations26 Nov
  3. 2020Denoising Diffusion Implicit Models6 Oct
  4. 2020Denoising Diffusion Probabilistic Models19 Jun

Other references · 93

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,950
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,304
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,588
  4. 2020Language Models are Few-Shot Learnerscited by 32,320
  5. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,043
  6. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,549
  7. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,476
  8. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,492
  9. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,364
  10. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,915
  11. 2018A Style-Based Generator Architecture for Generative Adversarial Networkscited by 6,000
  12. 2016Layer Normalizationcited by 5,624
38 more
  1. 2022Generative Adversarial Networkscited by 5,614
  2. 2022Flow Matching for Generative Modelingcited by 5,542
  3. 2017mixup: Beyond Empirical Risk Minimizationcited by 5,170
  4. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,730
  5. 2017Neural Discrete Representation Learningcited by 4,680
  6. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,628
  7. 2021Zero-Shot Text-to-Image Generationcited by 4,284
  8. 2019Generative Modeling by Estimating Gradients of the Data Distributioncited by 4,077
  9. 2020Taming Transformers for High-Resolution Image Synthesiscited by 3,237
  10. 2016Improved Techniques for Training GANscited by 2,946
  11. 2020Exploring Simple Siamese Representation Learningcited by 2,860
  12. 2023Diffusion Policy: Visuomotor Policy Learning via Action Diffusioncited by 2,760
  13. 2018Large Scale GAN Training for High Fidelity Natural Image Synthesiscited by 2,619
  14. 2021BEiT: BERT Pre-Training of Image Transformerscited by 2,368
  15. 2022Video Diffusion Modelscited by 2,293
  16. 2024CogVideoX: Text-to-Video Diffusion Models with An Expert Transformercited by 2,218
  17. 2024Qwen2 Technical Reportcited by 2,102
  18. 2021Barlow Twins: Self-Supervised Learning via Redundancy Reductioncited by 1,826
  19. 2021An Empirical Study of Training Self-Supervised Vision Transformerscited by 1,612
  20. 2018Unsupervised Representation Learning by Predicting Image Rotationscited by 1,590
  21. 2024Octo: An Open-Source Generalist Robot Policycited by 1,450
  22. 2016Context Encoders: Feature Learning by Inpaintingcited by 1,427
  23. 2016Unsupervised Learning of Visual Representations by Solving Jigsaw Puzzlescited by 1,381
  24. 2019Generating Diverse High-Fidelity Images with VQ-VAE-2cited by 1,381
  25. 2017Multi-Task Learning Using Uncertainty to Weigh Losses for Scene Geometry and Semanticscited by 1,348
  26. 2019Contrastive Multiview Codingcited by 1,297
  27. 2021SimMIM: A Simple Framework for Masked Image Modelingcited by 1,150
  28. 2020HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesiscited by 1,148
  29. 2022MaskGIT: Masked Generative Image Transformercited by 1,090
  30. 2021VICReg: Variance-Invariance-Covariance Regularization for Self-Supervised Learningcited by 1,033
  31. 2023Self-Supervised Learning from Images with a Joint-Embedding Predictive Architecturecited by 1,018
  32. 2017Sigmoid-Weighted Linear Units for Neural Network Function Approximation in Reinforcement Learningcited by 939
  33. 2019Improved Precision and Recall Metric for Assessing Generative Modelscited by 928
  34. 2023Stochastic Interpolants: A Unifying Framework for Flows and Diffusionscited by 917
  35. 2021iBOT: Image BERT Pre-Training with Online Tokenizercited by 895
  36. 2016Conditional Image Generation with PixelCNN Decoderscited by 861
  37. 2016Pixel Recurrent Neural Networkscited by 815
  38. 2018Self-Attention Generative Adversarial Networkscited by 809

50 most cited of 93 All 93 (JSON)

Built on by

Cited by year (2026 to 13 Oct)

Citing papers per year: 2024: 2; 2025: 4; 2026: 7.

13 papers.

Sort:

By area:

  1. 2024[MASK] is All You Needcited by 11
  2. 2025LayerSync: Self-aligning Intermediate Layerscited by 8
  3. 2025JEPA-Reasoner: Decoupling Latent Reasoning from Token Generationcited by 4
  4. 2024High-Resolution Image Synthesis via Next-Token Predictioncited by 3
  5. 2025Discrete JEPA: Learning Discrete Token Representations without Reconstructioncited by 2
  6. 2026From Alignment to Prediction: A Study of Self-Supervised Learning and Predictive Representation Learningcited by 2
  7. 2026Beyond Representation Learning: A Systematic Study of Joint-Embedding Predictive Generation for 3D Brain MRIcited by 1
  8. 2025JEPA-T: Joint-Embedding Predictive Architecture with Text Fusion for Image Generationcited by 0
  9. 2026AGE: Adaptive-masking for Graph Embedding in Graph Retrieval-Augmented Generationcited by 0
  10. 2026Optimizing ML Workload Partitioning between CPUs and CIM Accelerators for Heterogeneous Computingcited by 0
  11. 2026RepFlow: Reciprocal Supervision Improves Generation and Representation in Flow Modelscited by 0
  12. 2026One Latent, Many Tokens: Jointly Learning Compressed Embeddings for Efficient Language Diffusioncited by 0
1 more
  1. 2026EpicWorldModel: Exploration-driven Planning with Latent World Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Improving Joint Embedding Predictive Architecture with Diffusion Noise
  2. 2022Joint Embedding Predictive Architectures Focus on Slow Features
  3. 2026Embedding Prediction Helps Image Generation
  4. 2026Beyond Generative Priors: Minority Sampling with JEPA-Guided Diffusion
  5. 2026DLLM-JEPA: Joint Embedding Predictive Architectures for Masked Diffusion Language Models
  6. 2024Representation Alignment for Generation: Training Diffusion Transformers Is Easier Than You Think
  7. 2022Image Embedding for Denoising Generative Models
  8. 2023Conditional Generation from Unconditional Diffusion Models using Denoiser Representations