mlsanityData to 7 Oct 2026

CleanDIFT: Diffusion Features without Noise

arXiv 2412.03439 ↗First posted 4 Dec 2024cs.CVcited by 42

On the map: Diffusion models → Diffusion foundations and sampling

Abstract (arXiv)

Internal features from large-scale pre-trained diffusion models have recently been established as powerful semantic descriptors for a wide range of downstream tasks. Works that use these features generally need to add noise to images before passing them through the model to obtain the semantic features, as the models do not offer the most useful features when given images with little to no noise. We show that this noise has a critical impact on the usefulness of these features that cannot be remedied by ensembling with different random noises. We address this issue by introducing a lightweight, unsupervised fine-tuning method that enables diffusion backbones to provide high-quality, noise-free semantic features. We show that these features readily outperform previous diffusion features by a wide margin in a wide variety of extraction setups and downstream tasks, offering better performance than even ensemble-based methods at a fraction of the cost.

Key facts. Cited by 42 papers in the map (data through 7 Oct 2026). Placed in Diffusion foundations and sampling, one of 14 directions in Diffusion models. Builds on 53 of its 69 references that are in the map. First posted on arXiv 4 Dec 2024.

Builds on

53 of 69 references.

Closest ancestors · 14

Same direction, newest first

  1. 2024Diffusion Models and Representation Learning: A Survey30 Jun
  2. 2024Scaling Rectified Flow Transformers for High-Resolution Image Synthesis5 Mar
  3. 2024Dynamical Regimes of Diffusion Models28 Feb
  4. 2023SODA: Bottleneck Diffusion Models for Representation Learning29 Nov
  5. 2023Adversarial Diffusion Distillation28 Nov
  6. 2023SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesis4 Jul
  7. 2022Scalable Diffusion Models with Transformers19 Dec
  8. 2022On Distillation of Guided Diffusion Models6 Oct
  9. 2022Classifier-Free Diffusion Guidance26 Jul
  10. 2021High-Resolution Image Synthesis with Latent Diffusion Models20 Dec
  11. 2021Diffusion Models Beat GANs on Image Synthesis11 May
  12. 2020Score-Based Generative Modeling through Stochastic Differential Equations26 Nov
2 more
  1. 2020Denoising Diffusion Implicit Models6 Oct
  2. 2020Denoising Diffusion Probabilistic Models19 Jun

Other references · 39

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,604
  3. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  4. 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,764
  5. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,339
  6. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,893
  7. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,322
  8. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,131
  9. 2019Generative Modeling by Estimating Gradients of the Data Distributioncited by 4,067
  10. 2021GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Modelscited by 3,595
  11. 2017FiLM: Visual Reasoning with a General Conditioning Layercited by 2,286
  12. 2022Video Diffusion Modelscited by 2,280
27 more
  1. 2022Make-A-Video: Text-to-Video Generation without Text-Video Datacited by 1,725
  2. 2020GLU Variants Improve Transformercited by 1,724
  3. 2018Unsupervised Representation Learning by Predicting Image Rotationscited by 1,586
  4. 2019Root Mean Square Layer Normalizationcited by 1,425
  5. 2016Unsupervised Learning of Visual Representations by Solving Jigsaw Puzzlescited by 1,379
  6. 2020DiffWave: A Versatile Diffusion Model for Audio Synthesiscited by 1,339
  7. 2022eDiff-I: Text-to-Image Diffusion Models with an Ensemble of Expert Denoiserscited by 921
  8. 2023Vision Transformers Need Registerscited by 844
  9. 2019Self-Supervised Learning of Pretext-Invariant Representationscited by 806
  10. 2024Movie Gen: A Cast of Media Foundation Modelscited by 614
  11. 2020AdaBins: Depth Estimation using Adaptive Binscited by 596
  12. 2023Structure and Content-Guided Video Synthesis with Diffusion Modelscited by 589
  13. 2021Label-Efficient Semantic Segmentation with Diffusion Modelscited by 495
  14. 2022DiffusionDet: Diffusion Model for Object Detectioncited by 354
  15. 2023Your Diffusion Model is Secretly a Zero-Shot Classifiercited by 301
  16. 2023A Tale of Two Features: Stable Diffusion Complements DINO for Zero-Shot Semantic Correspondencecited by 297
  17. 2023Diffusion Hyperfeatures: Searching Through Time and Space for Semantic Correspondencecited by 211
  18. 2024Fast Timing-Conditioned Latent Audio Diffusioncited by 178
  19. 2022BinsFormer: Revisiting Adaptive Bins for Monocular Depth Estimationcited by 166
  20. 2022Generative Modelling With Inverse Heat Dissipationcited by 164
  21. 2023DiffuMask: Synthesizing Images with Pixel-level Annotations for Semantic Segmentation Using Diffusion Modelscited by 162
  22. 2019SPair-71k: A Large-scale Benchmark for Semantic Correspondencecited by 140
  23. 2023Unsupervised Semantic Correspondence Using Stable Diffusioncited by 120
  24. 2023Telling Left from Right: Identifying Geometry-Aware Semantic Correspondencecited by 85
  25. 2024EmerDiff: Emerging Pixel-level Semantic Knowledge in Diffusion Modelscited by 42
  26. 2023SD4Match: Learning to Prompt Stable Diffusion Model for Semantic Matchingcited by 38
  27. 2023DreamTeacher: Pretraining Image Backbones with Deep Generative Modelscited by 37

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 21; 2026: 20.

42 papers.

Sort:

By area:

  1. 2025Emergent Temporal Correspondences from Video Diffusion Transformerscited by 30
  2. 2025Understanding Representation Dynamics of Diffusion Models via Low-Dimensional Modelingcited by 18
  3. 2025Do It Yourself: Learning Semantic Correspondence from Pseudo-Labelscited by 13
  4. 2024[MASK] is All You Needcited by 11
  5. 2025Attention (as Discrete-Time Markov) Chainscited by 11
  6. 2025LayerSync: Self-aligning Intermediate Layerscited by 7
  7. 2025GECO: Geometrically Consistent Embedding with Lightspeed Inferencecited by 6
  8. 2025Mind-the-Glitch: Visual Correspondence for Detecting Inconsistencies in Subject-Driven Generationcited by 6
  9. 2025SCFlow: Implicitly Learning Style and Content Disentanglement with Flow Modelscited by 5
  10. 2025Studying Image Diffusion Features for Zero-Shot Video Object Segmentationcited by 4
  11. 2025Inverse Virtual Try-On: Generating Multi-Category Product-Style Images from Clothed Individualscited by 4
  12. 2025Semantic Correspondence: Unified Benchmarking and a Strong Baselinecited by 3
30 more
  1. 2025Towards Geometry-Grounded Dense Semantic Matching with VGGT Priorscited by 3
  2. 2025Probing and Leveraging Video Diffusion Transformer Features for Robust Point Trackingcited by 3
  3. 2026DDiT: Dynamic Patch Scheduling for Efficient Diffusion Transformerscited by 3
  4. 2026HeRO: Hierarchical 3D Semantic Representation for Pose-aware Object Manipulationcited by 3
  5. 2026GenMask: Adapting DiT for Segmentation via Direct Mask Generationcited by 3
  6. 2025Diffusion-Based Action Recognition Generalizes to Untrained Domainscited by 2
  7. 2025Latent Diffusion U-Net Representations Contain Positional Embeddings and Anomaliescited by 1
  8. 2025Heeding the Inner Voice: Aligning ControlNet Training via Intermediate Features Feedbackcited by 1
  9. 2026Control-DINO: Feature Space Conditioning for Controllable Image-to-Video Diffusioncited by 1
  10. 2026ForensicConcept: Transferable Forensic Concepts for AIGI Detectioncited by 1
  11. 2026Evaluating the Representation Space of Diffusion Models via Self-Supervised Principlescited by 1
  12. 2025StructDiff: Structure-aware Diffusion Model for 3D Fine-grained Medical Image Synthesiscited by 0
  13. 2025Annotation-Free One-Shot Imitation Learning for Multi-Step Manipulation Taskscited by 0
  14. 2025Rethinking Garment Conditioning in Diffusion-based Virtual Try-On: Decouple, Don't Denoisecited by 0
  15. 2025GeoDiffMM: Geometry-Guided Conditional Diffusion for Motion Magnificationcited by 0
  16. 2025DiffusionBrowser: Interactive Diffusion Previews via Multi-Branch Decoderscited by 0
  17. 2026Latent Structure Emergence in Diffusion Models via Confidence-Based Filteringcited by 0
  18. 2026Expanding the Role of Diffusion Models for Robust Classifier Trainingcited by 0
  19. 2026SOCO: Benchmarking Semantic Object Correspondence in Vision Foundation Modelscited by 0
  20. 2026DiT-Reward: Generative Representations for Text-to-Image Reward Modelingcited by 0
  21. 2026Self-Improving Diffusion Classifiers with Minority Preference Optimizationcited by 0
  22. 2026Promptable Animal Pose Tracking Across Speciescited by 0
  23. 2026DIFFCZSL: Compositional Zero-Shot Learning Regularized by Diffusion Representationscited by 0
  24. 2026ReaDiT Guidance: Control for Image and Video Generation using Diffusion Transformer Featurescited by 0
  25. 2026Diffuse2Seg: Diffusion Models Can Segment Anything Without Supervisioncited by 0
  26. 2026Geometry Without Coordinates: LiDAR Diffusion as a 3D Feature Bridgecited by 0
  27. 2026LG-VLN: A Zero-Shot Vision-and-Language Navigation Framework with LangGraph State Orchestrationcited by 0
  28. 2026Beyond Future Prediction: Denoising as Generative Adaptation for Robot Controlcited by 0
  29. 2026RepFlow: Reciprocal Supervision Improves Generation and Representation in Flow Modelscited by 0
  30. 2026Residual-Stream Burden Shapes Representation Learning in Diffusion Transformerscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Unsupervised Region-Based Image Editing of Denoising Diffusion Models
  2. 2026Should We Skip Diffusion?
  3. 2024Decoding Diffusion: A Scalable Framework for Unsupervised Analysis of Latent Space Biases and Representations Using Natural Language Prompts
  4. 2025Automated Learning of Semantic Embedding Representations for Diffusion Models
  5. 2023NoiseCLR: A Contrastive Learning Approach for Unsupervised Discovery of Interpretable Directions in Diffusion Models
  6. 2024textitRevelio: Interpreting and leveraging semantic information in diffusion models
  7. 2023Expanding Expressiveness of Diffusion Models with Limited Data via Self-Distillation based Fine-Tuning
  8. 2025Diffusion Models: A Mathematical Introduction