# Where's Waldo: Diffusion Features for Personalized Segmentation and Retrieval

arXiv: https://arxiv.org/abs/2405.18025 · HTML: https://mlsanity.com/paper/2405.18025/

## Key facts

- Cited by 18 papers in the map (data through 14 Oct 2026).
- Builds on 37 of its 46 references that are in the map.
- First posted on arXiv 28 May 2024.

## Builds on (most cited first, 30 of 37)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [Denoising Diffusion Implicit Models](https://mlsanity.com/paper/2010.02502/) (2020)
- [Segment Anything](https://mlsanity.com/paper/2304.02643/) (2023)
- [DINOv2: Learning Robust Visual Features without Supervision](https://mlsanity.com/paper/2304.07193/) (2023)
- [Masked Autoencoders Are Scalable Vision Learners](https://mlsanity.com/paper/2111.06377/) (2021)
- [Emerging Properties in Self-Supervised Vision Transformers](https://mlsanity.com/paper/2104.14294/) (2021)
- [Hierarchical Text-Conditional Image Generation with CLIP Latents](https://mlsanity.com/paper/2204.06125/) (2022)
- [Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding](https://mlsanity.com/paper/2205.11487/) (2022)
- [BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models](https://mlsanity.com/paper/2301.12597/) (2023)
- [eDiff-I: Text-to-Image Diffusion Models with an Ensemble of Expert Denoisers](https://mlsanity.com/paper/2211.01324/) (2022)
- [Blended Diffusion for Text-driven Editing of Natural Images](https://mlsanity.com/paper/2111.14818/) (2021)
- [Plug-and-Play Diffusion Features for Text-Driven Image-to-Image Translation](https://mlsanity.com/paper/2211.12572/) (2022)
- [LightGlue: Local Feature Matching at Light Speed](https://mlsanity.com/paper/2306.13643/) (2023)
- [Segment Everything Everywhere All at Once](https://mlsanity.com/paper/2304.06718/) (2023)
- [Emergent Correspondence from Image Diffusion](https://mlsanity.com/paper/2306.03881/) (2023)
- [SLIP: Self-supervision meets Language-Image Pre-training](https://mlsanity.com/paper/2112.12750/) (2021)
- [A Tale of Two Features: Stable Diffusion Complements DINO for Zero-Shot Semantic Correspondence](https://mlsanity.com/paper/2305.15347/) (2023)
- [Personalize Segment Anything Model with One Shot](https://mlsanity.com/paper/2305.03048/) (2023)
- [Splicing ViT Features for Semantic Appearance Transfer](https://mlsanity.com/paper/2201.00424/) (2022)
- [Revisiting Oxford and Paris: Large-Scale Image Retrieval Benchmarking](https://mlsanity.com/paper/1803.11285/) (2018)
- [Cross-Image Attention for Zero-Shot Appearance Transfer](https://mlsanity.com/paper/2311.03335/) (2023)
- [Diffuse, Attend, and Segment: Unsupervised Zero-Shot Segmentation using Stable Diffusion](https://mlsanity.com/paper/2308.12469/) (2023)
- [Training-Free Consistent Text-to-Image Generation](https://mlsanity.com/paper/2402.03286/) (2024)
- [MAST: A Memory-Augmented Self-supervised Tracker](https://mlsanity.com/paper/2002.07793/) (2020)
- [R2Former: Unified Retrieval and Reranking Transformer for Place Recognition](https://mlsanity.com/paper/2304.03410/) (2023)
- [Controllable Generation with Text-to-Image Diffusion Models: A Survey](https://mlsanity.com/paper/2403.04279/) (2024)
- [Effective Real Image Editing with Accelerated Iterative Diffusion Inversion](https://mlsanity.com/paper/2309.04907/) (2023)
- [SIGMA: Semantic-complete Graph Matching for Domain Adaptive Object Detection](https://mlsanity.com/paper/2203.06398/) (2022)

## Built on by (most cited first, 18 of 18)

- [From Image to Video: An Empirical Study of Diffusion Representations](https://mlsanity.com/paper/2502.07001/) (2025)
- [Personalized Vision via Visual In-Context Learning](https://mlsanity.com/paper/2509.25172/) (2025)
- [INSID3: Training-Free In-Context Segmentation with DINOv3](https://mlsanity.com/paper/2603.28480/) (2026)
- [Training-Free Personalization via Retrieval and Reasoning on Fingerprints](https://mlsanity.com/paper/2503.18623/) (2025)
- [Personalized Representation from Personalized Generation](https://mlsanity.com/paper/2412.16156/) (2024)
- [Teaching VLMs to Localize Specific Objects from In-context Examples](https://mlsanity.com/paper/2411.13317/) (2024)
- [Diffusion Models are Open-World Affordance Learners: Leveraging Generative Priors for 3D Affordance Learning](https://mlsanity.com/paper/2508.01651/) (2025)
- [FOCUS: Forcing In-Context Object Localization through Visual Support Constraints and Policy Optimization](https://mlsanity.com/paper/2605.31145/) (2026)
- [Toward Early Quality Assessment of Text-to-Image Diffusion Models](https://mlsanity.com/paper/2603.02829/) (2026)
- [Find your Needle: Small Object Image Retrieval via Multi-Object Attention Optimization](https://mlsanity.com/paper/2503.07038/) (2025)
- [ID-Sim: An Identity-Focused Similarity Metric](https://mlsanity.com/paper/2604.05039/) (2026)
- [Composed Object Retrieval: Object-level Retrieval via Composed Expressions](https://mlsanity.com/paper/2508.04424/) (2025)
- [Per-Query Visual Concept Learning](https://mlsanity.com/paper/2508.09045/) (2025)
- [FindIt: A Format-Informed Visual Detection Benchmark for Generalist Multimodal LLMs](https://mlsanity.com/paper/2606.04282/) (2026)
- [Personalizing MLLMs via Reinforced Multimodal Reference Game](https://mlsanity.com/paper/2606.28845/) (2026)
- [Personalized Object Identification and Localization via In-Context Inference with Vision-Language Models](https://mlsanity.com/paper/2607.00357/) (2026)
- [UniProbe: A Learnable Token-Level Hallucination Detector for Large VLMs using Multi-Structural Internal Representations](https://mlsanity.com/paper/2608.10835/) (2026)
- [FoundYou: A Unified Model for Personalized Segmentation and Retrieval](https://mlsanity.com/paper/2608.29917/) (2026)

## Abstract (arXiv)

Personalized retrieval and segmentation aim to locate specific instances within a dataset based on an input image and a short description of the reference instance. While supervised methods are effective, they require extensive labeled data for training. Recently, self-supervised foundation models have been introduced to these tasks showing comparable results to supervised methods. However, a significant flaw in these models is evident: they struggle to locate a desired instance when other instances within the same class are presented. In this paper, we explore text-to-image diffusion models for these tasks. Specifically, we propose a novel approach called PDM for Personalized Features Diffusion Matching, that leverages intermediate features of pre-trained text-to-image models for personalization tasks without any additional training. PDM demonstrates superior performance on popular retrieval and segmentation benchmarks, outperforming even supervised methods. We also highlight notable shortcomings in current instance and segmentation datasets and propose new benchmarks for these tasks.