mlsanityData to 14 Oct 2026

USP: Unified Self-Supervised Pretraining for Image Generation and Understanding

arXiv 2503.06132 ↗First posted 8 Mar 2025cs.CVcited by 24

On the map: Diffusion models → Diffusion foundations and sampling

Abstract (arXiv)

Recent studies have highlighted the interplay between diffusion models and representation learning. Intermediate representations from diffusion models can be leveraged for downstream visual tasks, while self-supervised vision models can enhance the convergence and generation quality of diffusion models. However, transferring pretrained weights from vision models to diffusion models is challenging due to input mismatches and the use of latent spaces. To address these challenges, we propose Unified Self-supervised Pretraining (USP), a framework that initializes diffusion models via masked latent modeling in a Variational Autoencoder (VAE) latent space. USP achieves comparable performance in understanding tasks while significantly improving the convergence speed and generation quality of diffusion models. Our code will be publicly available at https://github.com/AMAP-ML/USP.

Key facts. Cited by 24 papers in the map (data through 14 Oct 2026). Placed in Diffusion foundations and sampling, one of 14 directions in Diffusion models. Builds on 80 of its 99 references that are in the map. First posted on arXiv 8 Mar 2025.

Builds on

80 of 99 references.

Closest ancestors · 19

Same direction, newest first

  1. 2025Exploring Representation-Aligned Latent Space for Better Generation1 Feb
  2. 2025Reconstruction vs. Generation: Taming Optimization Dilemma in Latent Diffusion Models2 Jan
  3. 2024Representation Alignment for Generation: Training Diffusion Transformers Is Easier Than You Think9 Oct
  4. 2024Unified Auto-Encoding with Masked Diffusion25 Jun
  5. 2024Deconstructing Denoising Diffusion Models for Self-Supervised Learning25 Jan
  6. 2024SiT: Exploring Flow and Diffusion-based Generative Models with Scalable Interpolant Transformers16 Jan
  7. 2023Do text-free diffusion models learn discriminative visual representations?29 Nov
  8. 2023SODA: Bottleneck Diffusion Models for Representation Learning29 Nov
  9. 2023Diffusion Model as Representation Learner21 Aug
  10. 2023Fast Training of Diffusion Models with Masked Transformers15 Jun
  11. 2023Diffusion Models as Masked Autoencoders6 Apr
  12. 2023MDTv2: Masked Diffusion Transformer is a Strong Image Synthesizer25 Mar
7 more
  1. 2023Denoising Diffusion Autoencoders are Unified Self-supervised Learners17 Mar
  2. 2022Scalable Diffusion Models with Transformers19 Dec
  3. 2021High-Resolution Image Synthesis with Latent Diffusion Models20 Dec
  4. 2021Diffusion Models Beat GANs on Image Synthesis11 May
  5. 2020Score-Based Generative Modeling through Stochastic Differential Equations26 Nov
  6. 2020Denoising Diffusion Implicit Models6 Oct
  7. 2020Denoising Diffusion Probabilistic Models19 Jun

Other references · 61

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  3. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,139
  4. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 12,049
  5. 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,639
  6. 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,974
  7. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,513
  8. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,403
  9. 2017Mask R-CNNcited by 7,277
  10. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,968
  11. 2022Flow Matching for Generative Modelingcited by 5,702
  12. 2015Rethinking the Inception Architecture for Computer Visioncited by 5,475
38 more
  1. 2014Fully Convolutional Networks for Semantic Segmentationcited by 4,859
  2. 2015You Only Look Once: Unified, Real-Time Object Detectioncited by 4,852
  3. 2017Neural Discrete Representation Learningcited by 4,712
  4. 2014Going Deeper with Convolutionscited by 4,241
  5. 2022Flow Straight and Fast: Learning to Generate and Transfer Data with Rectified Flowcited by 3,700
  6. 2022A ConvNet for the 2020scited by 3,579
  7. 2020Taming Transformers for High-Resolution Image Synthesiscited by 3,253
  8. 2021SegFormer: Simple and Efficient Design for Semantic Segmentation with Transformerscited by 2,940
  9. 2019CutMix: Regularization Strategy to Train Strong Classifiers with Localizable Featurescited by 2,884
  10. 2019Benchmarking Neural Network Robustness to Common Corruptions and Perturbationscited by 2,877
  11. 2020Exploring Simple Siamese Representation Learningcited by 2,866
  12. 2020Unsupervised Learning of Visual Features by Contrasting Cluster Assignmentscited by 2,857
  13. 2021BEiT: BERT Pre-Training of Image Transformerscited by 2,375
  14. 2019RandAugment: Practical automated data augmentation with a reduced search spacecited by 2,269
  15. 2023Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyondcited by 2,224
  16. 2024Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scalingcited by 1,599
  17. 2019FCOS: Fully Convolutional One-Stage Object Detectioncited by 1,577
  18. 2020Big Self-Supervised Models are Strong Semi-Supervised Learnerscited by 1,380
  19. 2018Unified Perceptual Parsing for Scene Understandingcited by 1,271
  20. 2021SimMIM: A Simple Framework for Masked Image Modelingcited by 1,153
  21. 2016Deep Networks with Stochastic Depthcited by 979
  22. 2019Improved Precision and Recall Metric for Assessing Generative Modelscited by 944
  23. 2024Chameleon: Mixed-Modal Early-Fusion Foundation Modelscited by 927
  24. 2024Show-o: One Single Transformer to Unify Multimodal Understanding and Generationcited by 752
  25. 2024Autoregressive Image Generation without Vector Quantizationcited by 651
  26. 2021Masked Feature Prediction for Self-Supervised Visual Pre-Trainingcited by 576
  27. 2021Twins: Revisiting the Design of Spatial Attention in Vision Transformerscited by 565
  28. 2017Large Batch Training of Convolutional Networkscited by 535
  29. 2021Label-Efficient Semantic Segmentation with Diffusion Modelscited by 499
  30. 2024Janus: Decoupling Visual Encoding for Unified Multimodal Understanding and Generationcited by 457
  31. 2022Unified-IO: A Unified Model for Vision, Language, and Multi-Modal Taskscited by 421
  32. 2022DeiT III: Revenge of the ViTcited by 419
  33. 2021Conditional Positional Encodings for Vision Transformerscited by 406
  34. 2023InstaFlow: One Step is Enough for High-Quality Diffusion-Based Text-to-Image Generationcited by 395
  35. 2024SEED-X: Multimodal Models with Unified Multi-granularity Comprehension and Generationcited by 331
  36. 2023Your Diffusion Model is Secretly a Zero-Shot Classifiercited by 306
  37. 2022BEiT v2: Masked Image Modeling with Vector-Quantized Visual Tokenizerscited by 303
  38. 2023DreamLLM: Synergistic Multimodal Comprehension and Creationcited by 297

50 most cited of 61 All 61 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 14; 2026: 10.

24 papers.

Sort:

By area:

  1. 2025FLUX-Text: A Simple and Advanced Diffusion Transformer Baseline for Scene Text Editingcited by 46
  2. 2026DreamX-World 1.0: A General-Purpose Interactive World Modelcited by 40
  3. 2025Stochastic Self-Guidance for Training-Free Enhancement of Diffusion Modelscited by 36
  4. 2025AlignTok: Aligning Visual Foundation Encoders to Tokenizers for Diffusion Modelscited by 36
  5. 2025Omni-Effects: Unified and Spatially-Controllable Visual Effects Generationcited by 27
  6. 2025From Editor to Dense Geometry Estimatorcited by 24
  7. 2025There is No VAE: End-to-End Pixel-Space Generative Modeling via Self-Supervised Pre-trainingcited by 24
  8. 2025NarrLV: Towards a Comprehensive Narrative-Centric Evaluation for Long Video Generationcited by 22
  9. 2025Taming Preference Mode Collapse via Directional Decoupling Alignment in Diffusion Reinforcement Learningcited by 18
  10. 2026Everything in Its Place: Benchmarking Spatial Intelligence of Text-to-Image Modelscited by 16
  11. 2026SVG-EAR: Parameter-Free Linear Compensation for Sparse Video Generation via Error-aware Routingcited by 10
  12. 2025ImagerySearch: Adaptive Test-Time Search for Video Generation Beyond Semantic Dependency Constraintscited by 9
12 more
  1. 2026From Scale to Speed: Adaptive Test-Time Scaling for Image Editingcited by 8
  2. 2025Semantic Context Matters: Improving Conditioning for Autoregressive Modelscited by 7
  3. 2025Eevee: Towards Close-up High-resolution Video-based Virtual Try-oncited by 7
  4. 2025Taming Hallucinations: Boosting MLLMs' Video Understanding via Counterfactual Video Generationcited by 4
  5. 2026TP-Blend: Textual-Prompt Attention Pairing for Precise Object-Style Blending in Diffusion Modelscited by 4
  6. 2025Effective Probabilistic Time Series Forecasting with Fourier Adaptive Noise-Separated Diffusioncited by 3
  7. 2025End-to-End Multi-Modal Diffusion Mambacited by 3
  8. 2026Elucidating the SNR-t Bias of Diffusion Probabilistic Modelscited by 3
  9. 2026Extending One-Step Image Generation from Class Labels to Text via Discriminative Text Representationcited by 2
  10. 2026MeDUET: Disentangled Unified Pretraining for 3D Medical Image Synthesis and Analysiscited by 1
  11. 2026Elucidating Representation Degradation Problem in Diffusion Model Trainingcited by 0
  12. 2026A radiographic world model for clinical reasoning and evidence generationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Boosting Generative Image Modeling via Joint Image-Feature Synthesis
  2. 2025Diffusion Transformers with Representation Autoencoders
  3. 2025Masked Autoencoders Are Effective Tokenizers for Diffusion Models
  4. 2026UniGP: Taming Diffusion Transformer for Prior-Preserved Unified Generation and Perception
  5. 2022Exploring Vision Transformers as Diffusion Learners
  6. 2025Latent Diffusion Models with Masked AutoEncoders
  7. 2023Masked Diffusion Models Are Fast Distribution Learners
  8. 2026The Diffusion Encoder