mlsanityData to 14 Oct 2026

High Fidelity Visualization of What Your Self-Supervised Representation Knows About

arXiv 2112.09164 ↗First posted 16 Dec 2021cs.LG cs.AIcited by 66

Not on the map. Why?

Abstract (arXiv)

Discovering what is learned by neural networks remains a challenge. In self-supervised learning, classification is the most common task used to evaluate how good a representation is. However, relying only on such downstream task can limit our understanding of what information is retained in the representation of a given input. In this work, we showcase the use of a Representation Conditional Diffusion Model (RCDM) to visualize in data space the representations learned by self-supervised models. The use of RCDM is motivated by its ability to generate high-quality samples -- on par with state-of-the-art generative models -- while ensuring that the representations of those samples are faithful i.e. close to the one used for conditioning. By using RCDM to analyze self-supervised models, we are able to clearly show visually that i) SSL (backbone) representation are not invariant to the data augmentations they were trained with -- thus debunking an often restated but mistaken belief; ii) SSL post-projector embeddings appear indeed invariant to these data augmentation, along with many other data symmetries; iii) SSL representations appear more robust to small adversarial perturbation of their inputs than representations trained in a supervised manner; and iv) that SSL-trained representations exhibit an inherent structure that can be explored thanks to RCDM visualization and enables image manipulation.

Key facts. Cited by 66 papers in the map (data through 14 Oct 2026). Builds on 60 of its 79 references that are in the map. First posted on arXiv 16 Dec 2021.

Builds on

60 of 79 references.

References · 60

  1. 2017Attention Is All You Needcited by 60,361
  2. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  5. 2020Denoising Diffusion Probabilistic Modelscited by 21,076
  6. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,713
  7. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 12,049
  8. 2020Denoising Diffusion Implicit Modelscited by 9,781
  9. 2020Score-Based Generative Modeling through Stochastic Differential Equationscited by 9,122
  10. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,498
  11. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,255
  12. 2018Representation Learning with Contrastive Predictive Codingcited by 7,777
38 more
  1. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,513
  2. 2014Explaining and Harnessing Adversarial Examplescited by 7,364
  3. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,058
  4. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,968
  5. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,750
  6. 2016The Cityscapes Dataset for Semantic Urban Scene Understandingcited by 4,444
  7. 2019Generative Modeling by Estimating Gradients of the Data Distributioncited by 4,101
  8. 2021Improved Denoising Diffusion Probabilistic Modelscited by 3,761
  9. 2019Analyzing and Improving the Image Quality of StyleGANcited by 3,588
  10. 2016Improved Techniques for Training GANscited by 2,970
  11. 2020Exploring Simple Siamese Representation Learningcited by 2,866
  12. 2020Unsupervised Learning of Visual Features by Contrasting Cluster Assignmentscited by 2,857
  13. 2018Large Scale GAN Training for High Fidelity Natural Image Synthesiscited by 2,626
  14. 2013Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Mapscited by 2,371
  15. 2013Visualizing and Understanding Convolutional Networkscited by 2,290
  16. 2014Conditional Generative Adversarial Netscited by 2,162
  17. 2018Unsupervised Feature Learning via Non-Parametric Instance-level Discriminationcited by 1,845
  18. 2021Barlow Twins: Self-Supervised Learning via Redundancy Reductioncited by 1,838
  19. 2018Unsupervised Representation Learning by Predicting Image Rotationscited by 1,591
  20. 2016Unsupervised Learning of Visual Representations by Solving Jigsaw Puzzlescited by 1,382
  21. 2017SmoothGrad: removing noise by adding noisecited by 1,109
  22. 2017Deep Image Priorcited by 1,098
  23. 2021VICReg: Variance-Invariance-Covariance Regularization for Self-Supervised Learningcited by 1,057
  24. 2020Improved Techniques for Training Score-Based Generative Modelscited by 1,043
  25. 2017Deeper, Broader and Artier Domain Generalizationcited by 986
  26. 2016Conditional Image Generation with PixelCNN Decoderscited by 862
  27. 2016Pixel Recurrent Neural Networkscited by 816
  28. 2018Self-Attention Generative Adversarial Networkscited by 809
  29. 2019Self-Supervised Learning of Pretext-Invariant Representationscited by 808
  30. 2016Improving Variational Inference with Inverse Autoregressive Flowcited by 640
  31. 2014Understanding Deep Image Representations by Inverting Themcited by 403
  32. 2014On the Number of Linear Regions of Deep Neural Networkscited by 381
  33. 2016A Learned Representation For Artistic Stylecited by 341
  34. 2016Generating Images with Perceptual Similarity Metrics based on Deep Networkscited by 222
  35. 2020How Well Do Self-Supervised Models Transfer?cited by 213
  36. 2016Synthesizing the preferred inputs for neurons in neural networks via deep generator networkscited by 181
  37. 2016Grad-CAM: Why did you say that?cited by 161
  38. 2021Instance-Conditioned GANcited by 121

50 most cited of 60 All 60 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 14; 2023: 18; 2024: 14; 2025: 12; 2026: 8.

66 papers.

Sort:

By area:

  1. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,585
  2. 2023Self-Supervised Learning from Images with a Joint-Embedding Predictive Architecturecited by 1,039
  3. 2022Diffusion Models in Vision: A Surveycited by 1,015
  4. 2024Revisiting Feature Prediction for Learning Visual Representations from Videocited by 531
  5. 2022Masked Siamese Networks for Label-Efficient Learningcited by 302
  6. 2023A Cookbook of Self-Supervised Learningcited by 250
  7. 2024MetaMorph: Multimodal Understanding and Generation via Instruction Tuningcited by 206
  8. 2023Text Embeddings Reveal (Almost) As Much As Textcited by 192
  9. 2022RankMe: Assessing the downstream performance of pretrained self-supervised representations by their rankcited by 153
  10. 2022Contrastive and Non-Contrastive Self-Supervised Learning Recover Global and Local Spectral Embedding Methodscited by 149
  11. 2023Return of Unconditional Generation: A Self-supervised Representation Generation Methodcited by 98
  12. 2023What Do Self-Supervised Vision Transformers Learn?cited by 93
38 more
  1. 2023Language Model Inversioncited by 68
  2. 2022Pushing the limits of self-supervised ResNets: Can we outperform supervised learning without labels on ImageNet?cited by 67
  3. 2022The Hidden Uniform Cluster Prior in Self-Supervised Learningcited by 58
  4. 2023Improving Visual Prompt Tuning for Self-supervised Vision Transformerscited by 57
  5. 2024Learning by Reconstruction Produces Uninformative Features For Perceptioncited by 55
  6. 2023A-JEPA: Joint-Embedding Predictive Architecture Can Listencited by 53
  7. 2022ImageNet-X: Understanding Model Mistakes with Factor of Variation Annotationscited by 44
  8. 2022Self-Guided Diffusion Modelscited by 37
  9. 2023Feedback-guided Data Synthesis for Imbalanced Classificationcited by 33
  10. 2023Identifying Interpretable Subspaces in Image Representationscited by 32
  11. 2025AnyUp: Universal Feature Upsamplingcited by 28
  12. 2022Guillotine Regularization: Why removing layers is needed to improve generalization in Self-Supervised Learningcited by 26
  13. 2024Transferable Embedding Inversion Attack: Uncovering Privacy Risks in Text Embeddings without Model Queriescited by 25
  14. 2023Improved baselines for vision-language pre-trainingcited by 20
  15. 2023Do SSL Models Have D'ej`a Vu? A Case of Unintended Memorization in Self-supervised Learningcited by 19
  16. 2025CheXWorld: Exploring Image World Modeling for Radiograph Representation Learningcited by 19
  17. 2023Towards Democratizing Joint-Embedding Self-Supervised Learningcited by 18
  18. 2025Missing Target-Relevant Information Prediction with World Model for Accurate Zero-Shot Composed Image Retrievalcited by 18
  19. 2025Highly Compressed Tokenizer Can Generate Without Trainingcited by 18
  20. 2024DisDet: Exploring Detectability of Backdoor Attack on Diffusion Modelscited by 17
  21. 2024Explaining generative diffusion models via visual analysis for interpretable decision-making processcited by 17
  22. 2023Objectives Matter: Understanding the Impact of Self-Supervised Objectives on Vision Transformer Representationscited by 16
  23. 2022The Robustness Limits of SoTA Vision Models to Natural Variationcited by 15
  24. 2025EchoWorld: Learning Motion-Aware World Models for Echocardiography Probe Guidancecited by 15
  25. 2022Bag of Image Patch Embedding Behind the Success of Self-Supervised Learningcited by 10
  26. 2023A Simple Recipe for Competitive Low-compute Self supervised Vision Modelscited by 10
  27. 2022Minimalistic Unsupervised Learning with the Sparse Manifold Transformcited by 6
  28. 2024Rethinking cluster-conditioned diffusion models for label-free image synthesiscited by 5
  29. 2022Measuring Self-Supervised Representation Quality for Downstream Classification using Discriminative Featurescited by 4
  30. 2023A surprisingly simple technique to control the pretraining bias for better transfer: Expand or Narrow your representationcited by 4
  31. 2024Can Generative Models Improve Self-Supervised Representation Learning?cited by 3
  32. 2024Review of Deep Representation Learning Techniques for Brain-Computer Interfaces and Recommendationscited by 3
  33. 2025Compositional Discrete Latent Code for High Fidelity, Productive Diffusion Modelscited by 3
  34. 2025High-Performance Self-Supervised Learning by Joint Training of Flow Matchingcited by 3
  35. 2023Self-Supervision for Tackling Unsupervised Anomaly Detection: Pitfalls and Opportunitiescited by 2
  36. 2024Large-vocabulary forensic pathological analyses via prototypical cross-modal contrastive learningcited by 2
  37. 2025AI Alignment in Medical Imaging: Unveiling Hidden Biases Through Counterfactual Analysiscited by 2
  38. 2025FeatInv: Spatially resolved mapping from feature space to input space using conditional diffusion modelscited by 2

50 most cited of 66 All 66 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026Evaluating the Representation Space of Diffusion Models via Self-Supervised Principles
  2. 2026Self-Supervised Representation-Guided Generative Dataset Distillation
  3. 2025No Other Representation Component Is Needed: Diffusion Transformers Can Provide Representation Guidance by Themselves
  4. 2026Self-Supervised Learning from Structural Invariance
  5. 2024Diffusion Models and Representation Learning: A Survey
  6. 2023Learned representation-guided diffusion models for large-image generation
  7. 2026Representation Learning in Diffusion and Flow-based Model: An Application Aspect
  8. 2025Canonical Latent Representations in Conditional Diffusion Models