# DreamSim: Learning New Dimensions of Human Visual Similarity using Synthetic Data

arXiv: https://arxiv.org/abs/2306.09344 · HTML: https://mlsanity.com/paper/2306.09344/

## Key facts

- Cited by 337 papers in the map (data through 14 Oct 2026).
- Builds on 52 of its 88 references that are in the map.
- First posted on arXiv 15 Jun 2023.

## Builds on (most cited first, 30 of 52)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [The Unreasonable Effectiveness of Deep Features as a Perceptual Metric](https://mlsanity.com/paper/1801.03924/) (2018)
- [Diffusion Models Beat GANs on Image Synthesis](https://mlsanity.com/paper/2105.05233/) (2021)
- [Momentum Contrast for Unsupervised Visual Representation Learning](https://mlsanity.com/paper/1911.05722/) (2019)
- [Masked Autoencoders Are Scalable Vision Learners](https://mlsanity.com/paper/2111.06377/) (2021)
- [Emerging Properties in Self-Supervised Vision Transformers](https://mlsanity.com/paper/2104.14294/) (2021)
- [Hierarchical Text-Conditional Image Generation with CLIP Latents](https://mlsanity.com/paper/2204.06125/) (2022)
- [Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding](https://mlsanity.com/paper/2205.11487/) (2022)
- [A Style-Based Generator Architecture for Generative Adversarial Networks](https://mlsanity.com/paper/1812.04948/) (2018)
- [Generative Adversarial Networks](https://mlsanity.com/paper/2203.00667/) (2022)
- [Perceptual Losses for Real-Time Style Transfer and Super-Resolution](https://mlsanity.com/paper/1603.08155/) (2016)
- [InstructPix2Pix: Learning to Follow Image Editing Instructions](https://mlsanity.com/paper/2211.09800/) (2022)
- [The Many Faces of Robustness: A Critical Analysis of Out-of-Distribution Generalization](https://mlsanity.com/paper/2006.16241/) (2020)
- [Towards Robust Monocular Depth Estimation: Mixing Datasets for Zero-shot Cross-dataset Transfer](https://mlsanity.com/paper/1907.01341/) (2019)
- [Contrastive Multiview Coding](https://mlsanity.com/paper/1906.05849/) (2019)
- [Semantic Image Synthesis with Spatially-Adaptive Normalization](https://mlsanity.com/paper/1903.07291/) (2019)
- [Depth Map Prediction from a Single Image using a Multi-Scale Deep Network](https://mlsanity.com/paper/1406.2283/) (2014)
- [Reproducible scaling laws for contrastive language-image learning](https://mlsanity.com/paper/2212.07143/) (2022)
- [Deep Image Prior](https://mlsanity.com/paper/1711.10925/) (2017)
- [Image Quality Assessment: Unifying Structure and Texture Similarity](https://mlsanity.com/paper/2004.07728/) (2020)
- [Scaling Vision Transformers to 22 Billion Parameters](https://mlsanity.com/paper/2302.05442/) (2023)
- [Muse: Text-To-Image Generation via Masked Generative Transformers](https://mlsanity.com/paper/2301.00704/) (2023)
- [Transformer Interpretability Beyond Attention Visualization](https://mlsanity.com/paper/2012.09838/) (2020)
- [Zero-Shot Text-Guided Object Generation with Dream Fields](https://mlsanity.com/paper/2112.01455/) (2021)

## Built on by (most cited first, 30 of 337)

- [Navigation World Models](https://mlsanity.com/paper/2412.03572/) (2024)
- [Adjoint Matching: Fine-tuning Flow and Diffusion Generative Models with Memoryless Stochastic Optimal Control](https://mlsanity.com/paper/2409.08861/) (2024)
- [VLM2Vec: Training Vision-Language Models for Massive Multimodal Embedding Tasks](https://mlsanity.com/paper/2410.05160/) (2024)
- [FlowEdit: Inversion-Free Text-Based Editing Using Pre-Trained Flow Models](https://mlsanity.com/paper/2412.08629/) (2024)
- [MANTIS: Interleaved Multi-Image Instruction Tuning](https://mlsanity.com/paper/2405.01483/) (2024)
- [Diffusion Model-Based Image Editing: A Survey](https://mlsanity.com/paper/2402.17525/) (2024)
- [GME: Improving Universal Multimodal Retrieval by Multimodal LLMs](https://mlsanity.com/paper/2412.16855/) (2024)
- [MM-Embed: Universal Multimodal Retrieval with Multimodal LLMs](https://mlsanity.com/paper/2411.02571/) (2024)
- [Getting aligned on representational alignment](https://mlsanity.com/paper/2310.13018/) (2023)
- [DOCCI: Descriptions of Connected and Contrasting Images](https://mlsanity.com/paper/2404.19753/) (2024)
- [DreamBench++: A Human-Aligned Benchmark for Personalized Image Generation](https://mlsanity.com/paper/2406.16855/) (2024)
- [State of the Art on Diffusion Models for Visual Computing](https://mlsanity.com/paper/2310.07204/) (2023)
- [URDFormer: A Pipeline for Constructing Articulated Simulation Environments from Real-World Images](https://mlsanity.com/paper/2405.11656/) (2024)
- [GenAI Arena: An Open Evaluation Platform for Generative Models](https://mlsanity.com/paper/2406.04485/) (2024)
- [VisualQuality-R1: Reasoning-Induced Image Quality Assessment via Reinforcement Learning to Rank](https://mlsanity.com/paper/2505.14460/) (2025)
- [ImagenHub: Standardizing the evaluation of conditional image generation models](https://mlsanity.com/paper/2310.01596/) (2023)
- [MM1.5: Methods, Analysis & Insights from Multimodal LLM Fine-tuning](https://mlsanity.com/paper/2409.20566/) (2024)
- [Vid2World: Crafting Video Diffusion Models to Interactive World Models](https://mlsanity.com/paper/2505.14357/) (2025)
- [VAGEN: Reinforcing World Model Reasoning for Multi-Turn VLM Agents](https://mlsanity.com/paper/2510.16907/) (2025)
- [RealFill: Reference-Driven Generation for Authentic Image Completion](https://mlsanity.com/paper/2309.16668/) (2023)
- [Readout Guidance: Learning Control from Diffusion Features](https://mlsanity.com/paper/2312.02150/) (2023)
- [Follow-Your-Click: Open-domain Regional Image Animation via Short Prompts](https://mlsanity.com/paper/2403.08268/) (2024)
- [SoK: Watermarking for AI-Generated Content](https://mlsanity.com/paper/2411.18479/) (2024)
- [World Action Models: The Next Frontier in Embodied AI](https://mlsanity.com/paper/2605.12090/) (2026)
- [DreamStory: Open-Domain Story Visualization by LLM-Guided Multi-Subject Consistent Diffusion](https://mlsanity.com/paper/2407.12899/) (2024)
- [AquaLoRA: Toward White-box Protection for Customized Stable Diffusion Models via Watermark LoRA](https://mlsanity.com/paper/2405.11135/) (2024)
- [VBench++: Comprehensive and Versatile Benchmark Suite for Video Generative Models](https://mlsanity.com/paper/2411.13503/) (2024)
- [WBench: A Comprehensive Multi-turn Benchmark for Interactive Video World Model Evaluation](https://mlsanity.com/paper/2605.25874/) (2026)
- [Amodal Completion via Progressive Mixed Context Diffusion](https://mlsanity.com/paper/2312.15540/) (2023)
- [FakeBench: Probing Explainable Fake Image Detection via Large Multimodal Models](https://mlsanity.com/paper/2404.13306/) (2024)

## Abstract (arXiv)

Current perceptual similarity metrics operate at the level of pixels and patches. These metrics compare images in terms of their low-level colors and textures, but fail to capture mid-level similarities and differences in image layout, object pose, and semantic content. In this paper, we develop a perceptual metric that assesses images holistically. Our first step is to collect a new dataset of human similarity judgments over image pairs that are alike in diverse ways. Critical to this dataset is that judgments are nearly automatic and shared by all observers. To achieve this we use recent text-to-image models to create synthetic pairs that are perturbed along various dimensions. We observe that popular perceptual metrics fall short of explaining our new data, and we introduce a new metric, DreamSim, tuned to better align with human perception. We analyze how our metric is affected by different visual attributes, and find that it focuses heavily on foreground objects and semantic content while also being sensitive to color and layout. Notably, despite being trained on synthetic data, our metric generalizes to real images, giving strong results on retrieval and reconstruction tasks. Furthermore, our metric outperforms both prior learned metrics and recent large vision models on these tasks.