mlsanityData to 14 Oct 2026

DreamSim: Learning New Dimensions of Human Visual Similarity using Synthetic Data

arXiv 2306.09344 ↗First posted 15 Jun 2023cs.CV cs.LGcited by 337

Not on the map. Why?

Cited by 74 in Diffusion models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Current perceptual similarity metrics operate at the level of pixels and patches. These metrics compare images in terms of their low-level colors and textures, but fail to capture mid-level similarities and differences in image layout, object pose, and semantic content. In this paper, we develop a perceptual metric that assesses images holistically. Our first step is to collect a new dataset of human similarity judgments over image pairs that are alike in diverse ways. Critical to this dataset is that judgments are nearly automatic and shared by all observers. To achieve this we use recent text-to-image models to create synthetic pairs that are perturbed along various dimensions. We observe that popular perceptual metrics fall short of explaining our new data, and we introduce a new metric, DreamSim, tuned to better align with human perception. We analyze how our metric is affected by different visual attributes, and find that it focuses heavily on foreground objects and semantic content while also being sensitive to color and layout. Notably, despite being trained on synthetic data, our metric generalizes to real images, giving strong results on retrieval and reconstruction tasks. Furthermore, our metric outperforms both prior learned metrics and recent large vision models on these tasks.

Key facts. Cited by 337 papers in the map (data through 14 Oct 2026). Builds on 52 of its 88 references that are in the map. First posted on arXiv 15 Jun 2023.

Builds on

52 of 88 references.

References · 52

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  3. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  4. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  5. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,364
  6. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  7. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 12,049
  8. 2018The Unreasonable Effectiveness of Deep Features as a Perceptual Metriccited by 11,177
  9. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,255
  10. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,513
  11. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,403
  12. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,968
38 more
  1. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,585
  2. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,355
  3. 2018A Style-Based Generator Architecture for Generative Adversarial Networkscited by 6,013
  4. 2022Generative Adversarial Networkscited by 5,627
  5. 2016Perceptual Losses for Real-Time Style Transfer and Super-Resolutioncited by 2,694
  6. 2022InstructPix2Pix: Learning to Follow Image Editing Instructionscited by 2,611
  7. 2020The Many Faces of Robustness: A Critical Analysis of Out-of-Distribution Generalizationcited by 1,850
  8. 2019Towards Robust Monocular Depth Estimation: Mixing Datasets for Zero-shot Cross-dataset Transfercited by 1,495
  9. 2019Contrastive Multiview Codingcited by 1,301
  10. 2019Semantic Image Synthesis with Spatially-Adaptive Normalizationcited by 1,299
  11. 2014Depth Map Prediction from a Single Image using a Multi-Scale Deep Networkcited by 1,274
  12. 2022Reproducible scaling laws for contrastive language-image learningcited by 1,173
  13. 2017Deep Image Priorcited by 1,098
  14. 2020Image Quality Assessment: Unifying Structure and Texture Similaritycited by 811
  15. 2023Scaling Vision Transformers to 22 Billion Parameterscited by 694
  16. 2023Muse: Text-To-Image Generation via Masked Generative Transformerscited by 648
  17. 2020Transformer Interpretability Beyond Attention Visualizationcited by 627
  18. 2021Zero-Shot Text-Guided Object Generation with Dream Fieldscited by 513
  19. 2015A Neural Algorithm of Artistic Stylecited by 483
  20. 2021Putting NeRF on a Diet: Semantically Consistent Few-Shot View Synthesiscited by 464
  21. 2014Understanding Deep Image Representations by Inverting Themcited by 403
  22. 2019Making Convolutional Networks Shift-Invariant Againcited by 391
  23. 2020StyleSpace Analysis: Disentangled Controls for StyleGAN Image Generationcited by 332
  24. 2021Deep ViT Features as Dense Visual Descriptorscited by 330
  25. 2023Synthetic Data from Diffusion Models Improves ImageNet Classificationcited by 327
  26. 2022Is synthetic data from generative models ready for image recognition?cited by 320
  27. 2016Learning from Simulated and Unsupervised Images through Adversarial Trainingcited by 288
  28. 2016Generating Images with Perceptual Similarity Metrics based on Deep Networkscited by 222
  29. 2022Splicing ViT Features for Semantic Appearance Transfercited by 209
  30. 2022Fake it till you make it: Learning transferable representations from synthetic ImageNet clonescited by 172
  31. 2018PieAPP: Perceptual Image-Error Assessment through Pairwise Preferencecited by 160
  32. 2022Learning to generate line drawings that convey geometry and semanticscited by 83
  33. 2020StyleGAN2 Distillation for Feed-forward Image Manipulationcited by 78
  34. 2022Human alignment of neural network representationscited by 74
  35. 2021GAN-Supervised Dense Visual Alignmentcited by 62
  36. 2019E-LPIPS: Robust Perceptual Image Similarity via Random Transformation Ensemblescited by 45
  37. 2021CDPAM: Contrastive learning for perceptual audio similaritycited by 42
  38. 2022Do better ImageNet classifiers assess perceptual similarity better?cited by 34

50 most cited of 52 All 52 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 17; 2024: 64; 2025: 119; 2026: 137.

337 papers.

Sort:

By area:

  1. 2024Navigation World Modelscited by 272
  2. 2024Adjoint Matching: Fine-tuning Flow and Diffusion Generative Models with Memoryless Stochastic Optimal Controlcited by 233
  3. 2024VLM2Vec: Training Vision-Language Models for Massive Multimodal Embedding Taskscited by 228
  4. 2024FlowEdit: Inversion-Free Text-Based Editing Using Pre-Trained Flow Modelscited by 217
  5. 2024MANTIS: Interleaved Multi-Image Instruction Tuningcited by 216
  6. 2024Diffusion Model-Based Image Editing: A Surveycited by 212
  7. 2024GME: Improving Universal Multimodal Retrieval by Multimodal LLMscited by 170
  8. 2024MM-Embed: Universal Multimodal Retrieval with Multimodal LLMscited by 161
  9. 2023Getting aligned on representational alignmentcited by 143
  10. 2024DOCCI: Descriptions of Connected and Contrasting Imagescited by 132
  11. 2024DreamBench++: A Human-Aligned Benchmark for Personalized Image Generationcited by 131
  12. 2023State of the Art on Diffusion Models for Visual Computingcited by 127
38 more
  1. 2024URDFormer: A Pipeline for Constructing Articulated Simulation Environments from Real-World Imagescited by 111
  2. 2024GenAI Arena: An Open Evaluation Platform for Generative Modelscited by 82
  3. 2025VisualQuality-R1: Reasoning-Induced Image Quality Assessment via Reinforcement Learning to Rankcited by 79
  4. 2023ImagenHub: Standardizing the evaluation of conditional image generation modelscited by 77
  5. 2024MM1.5: Methods, Analysis & Insights from Multimodal LLM Fine-tuningcited by 76
  6. 2025Vid2World: Crafting Video Diffusion Models to Interactive World Modelscited by 76
  7. 2025VAGEN: Reinforcing World Model Reasoning for Multi-Turn VLM Agentscited by 62
  8. 2023RealFill: Reference-Driven Generation for Authentic Image Completioncited by 59
  9. 2023Readout Guidance: Learning Control from Diffusion Featurescited by 52
  10. 2024Follow-Your-Click: Open-domain Regional Image Animation via Short Promptscited by 52
  11. 2024SoK: Watermarking for AI-Generated Contentcited by 51
  12. 2026World Action Models: The Next Frontier in Embodied AIcited by 50
  13. 2024DreamStory: Open-Domain Story Visualization by LLM-Guided Multi-Subject Consistent Diffusioncited by 47
  14. 2024AquaLoRA: Toward White-box Protection for Customized Stable Diffusion Models via Watermark LoRAcited by 46
  15. 2024VBench++: Comprehensive and Versatile Benchmark Suite for Video Generative Modelscited by 45
  16. 2026WBench: A Comprehensive Multi-turn Benchmark for Interactive Video World Model Evaluationcited by 39
  17. 2023Amodal Completion via Progressive Mixed Context Diffusioncited by 38
  18. 2024FakeBench: Probing Explainable Fake Image Detection via Large Multimodal Modelscited by 36
  19. 2024Consistency-diversity-realism Pareto fronts of conditional image generative modelscited by 36
  20. 2023Does CLIP's Generalization Performance Mainly Stem from High Train-Test Similarity?cited by 35
  21. 2023Retrieval-Augmented Layout Transformer for Content-Aware Layout Generationcited by 35
  22. 2024PosterLlama: Bridging Design Ability of Langauge Model to Contents-Aware Layout Generationcited by 35
  23. 2025PAI-Bench: A Comprehensive Benchmark For Physical AIcited by 34
  24. 2024Identity Decoupling for Multi-Subject Personalization of Text-to-Image Modelscited by 33
  25. 2025OmniConsistency: Learning Style-Agnostic Consistency from Paired Stylization Datacited by 32
  26. 2025Does FLUX Already Know How to Perform Physically Plausible Image Composition?cited by 32
  27. 2026Improved Baselines with Representation Autoencoderscited by 31
  28. 2024VideoGigaGAN: Towards Detail-rich Video Super-Resolutioncited by 30
  29. 2025ObjectMover: Generative Object Movement with Video Priorcited by 30
  30. 2025Whole-Body Conditioned Egocentric Video Predictioncited by 30
  31. 2025Learning 3D Persistent Embodied World Modelscited by 29
  32. 2026ABot-PhysWorld: Interactive World Foundation Model for Robotic Manipulation with Physics Alignmentcited by 29
  33. 2024EVA: An Embodied World Model for Future Video Anticipationcited by 28
  34. 2025SliderSpace: Decomposing the Visual Capabilities of Diffusion Modelscited by 26
  35. 2025Modality Curation: Building Universal Embeddings for Advanced Multimodal Information Retrievalcited by 26
  36. 2025Trust Region Constrained Measure Transport in Path Space for Stochastic Optimal Control and Inferencecited by 23
  37. 2025Distilling Diversity and Control in Diffusion Modelscited by 22
  38. 2025ShapeGen4D: Towards High Quality 4D Shape Generation from Videoscited by 21

50 most cited of 337 All 337 (JSON)