# GeneCIS: A Benchmark for General Conditional Image Similarity

arXiv: https://arxiv.org/abs/2306.07969 · HTML: https://mlsanity.com/paper/2306.07969/

## Key facts

- Cited by 47 papers in the map (data through 14 Oct 2026).
- Builds on 59 of its 89 references that are in the map.
- First posted on arXiv 13 Jun 2023.

## Builds on (most cited first, 30 of 59)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Representation Learning with Contrastive Predictive Coding](https://mlsanity.com/paper/1807.03748/) (2018)
- [Emerging Properties in Self-Supervised Vision Transformers](https://mlsanity.com/paper/2104.14294/) (2021)
- [LAION-5B: An open large-scale dataset for training next generation image-text models](https://mlsanity.com/paper/2210.08402/) (2022)
- [Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision](https://mlsanity.com/paper/2102.05918/) (2021)
- [GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models](https://mlsanity.com/paper/2112.10741/) (2021)
- [Unsupervised Learning of Visual Features by Contrasting Cluster Assignments](https://mlsanity.com/paper/2006.09882/) (2020)
- [Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotations](https://mlsanity.com/paper/1602.07332/) (2016)
- [InstructPix2Pix: Learning to Follow Image Editing Instructions](https://mlsanity.com/paper/2211.09800/) (2022)
- [FiLM: Visual Reasoning with a General Conditioning Layer](https://mlsanity.com/paper/1709.07871/) (2017)
- [Improved Baselines with Momentum Contrastive Learning](https://mlsanity.com/paper/2003.04297/) (2020)
- [Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Models](https://mlsanity.com/paper/1505.04870/) (2015)
- [LVIS: A Dataset for Large Vocabulary Instance Segmentation](https://mlsanity.com/paper/1908.03195/) (2019)
- [StyleCLIP: Text-Driven Manipulation of StyleGAN Imagery](https://mlsanity.com/paper/2103.17249/) (2021)
- [Blended Diffusion for Text-driven Editing of Natural Images](https://mlsanity.com/paper/2111.14818/) (2021)
- [Language-driven Semantic Segmentation](https://mlsanity.com/paper/2201.03546/) (2022)
- [DiffusionCLIP: Text-Guided Diffusion Models for Robust Image Manipulation](https://mlsanity.com/paper/2110.02711/) (2021)
- [Detecting Twenty-thousand Classes using Image-level Supervision](https://mlsanity.com/paper/2201.02605/) (2022)
- [Simple Open-Vocabulary Object Detection with Vision Transformers](https://mlsanity.com/paper/2205.06230/) (2022)
- [VQGAN-CLIP: Open Domain Image Generation and Editing with Natural Language Guidance](https://mlsanity.com/paper/2204.08583/) (2022)
- [Image Retrieval on Real-life Images with Pre-trained Vision-and-Language Models](https://mlsanity.com/paper/2108.04024/) (2021)
- [Composing Text and Image for Image Retrieval - An Empirical Odyssey](https://mlsanity.com/paper/1812.07119/) (2018)

## Built on by (most cited first, 30 of 47)

- [MuirBench: A Comprehensive Benchmark for Robust Multi-image Understanding](https://mlsanity.com/paper/2406.09411/) (2024)
- [MagicLens: Self-Supervised Image Retrieval with Open-Ended Instructions](https://mlsanity.com/paper/2403.19651/) (2024)
- [LamRA: Large Multimodal Model as Your Advanced Retrieval Assistant](https://mlsanity.com/paper/2412.01720/) (2024)
- [Vision-by-Language for Training-Free Compositional Image Retrieval](https://mlsanity.com/paper/2310.09291/) (2023)
- [No Representation Rules Them All in Category Discovery](https://mlsanity.com/paper/2311.17055/) (2023)
- [Reason-before-Retrieve: One-Stage Reflective Chain-of-Thoughts for Training-Free Zero-Shot Composed Image Retrieval](https://mlsanity.com/paper/2412.11077/) (2024)
- [Language-only Efficient Training of Zero-shot Composed Image Retrieval](https://mlsanity.com/paper/2312.01998/) (2023)
- [EgoCVR: An Egocentric Benchmark for Fine-Grained Composed Video Retrieval](https://mlsanity.com/paper/2407.16658/) (2024)
- [Are These the Same Apple? Comparing Images Based on Object Intrinsics](https://mlsanity.com/paper/2311.00750/) (2023)
- [Missing Target-Relevant Information Prediction with World Model for Accurate Zero-Shot Composed Image Retrieval](https://mlsanity.com/paper/2503.17109/) (2025)
- [A Comprehensive Survey on Composed Image Retrieval](https://mlsanity.com/paper/2502.18495/) (2025)
- [Pretrain like Your Inference: Masked Tuning Improves Zero-Shot Composed Image Retrieval](https://mlsanity.com/paper/2311.07622/) (2023)
- [U-MARVEL: Unveiling Key Factors for Universal Multimodal Retrieval via Embedding Learning with MLLMs](https://mlsanity.com/paper/2507.14902/) (2025)
- [IDMR: Towards Instance-Driven Precise Visual Correspondence in Multimodal Retrieval](https://mlsanity.com/paper/2504.00954/) (2025)
- [Scale Up Composed Image Retrieval Learning via Modification Text Generation](https://mlsanity.com/paper/2504.05316/) (2025)
- [TRACE: Task-Adaptive Reasoning and Representation Learning for Universal Multimodal Retrieval](https://mlsanity.com/paper/2603.02929/) (2026)
- [Joint Fusion and Encoding: Advancing Multimodal Retrieval from the Ground Up](https://mlsanity.com/paper/2502.20008/) (2025)
- [Conditional Representation Learning for Customized Tasks](https://mlsanity.com/paper/2510.04564/) (2025)
- [FocalLens: Instruction Tuning Enables Zero-Shot Conditional Image Representations](https://mlsanity.com/paper/2504.08368/) (2025)
- [DetailFusion: A Dual-branch Framework with Detail Enhancement for Composed Image Retrieval](https://mlsanity.com/paper/2505.17796/) (2025)
- [Steerable Visual Representations](https://mlsanity.com/paper/2604.02327/) (2026)
- [Revisiting Relevance Feedback for CLIP-based Interactive Image Retrieval](https://mlsanity.com/paper/2404.16398/) (2024)
- [Unveiling the Mystery of Visual Attributes of Concrete and Abstract Concepts: Variability, Nearest Neighbors, and Challenging Categories](https://mlsanity.com/paper/2410.11657/) (2024)
- [The Many Senses of Visual Similarity: A Text-Prompted Image Perceptual Metric](https://mlsanity.com/paper/2607.18237/) (2026)
- [An Efficient Post-hoc Framework for Reducing Task Discrepancy of Text Encoders for Composed Image Retrieval](https://mlsanity.com/paper/2406.09188/) (2024)
- [Modality and Task Adaptation for Enhanced Zero-shot Composed Image Retrieval](https://mlsanity.com/paper/2410.23736/) (2024)
- [Composed Multi-modal Retrieval: A Survey of Approaches and Applications](https://mlsanity.com/paper/2503.01334/) (2025)
- [V-Retrver: Evidence-Driven Agentic Reasoning for Universal Multimodal Retrieval](https://mlsanity.com/paper/2602.06034/) (2026)
- [G-MIXER: Geodesic Mixup-based Implicit Semantic Expansion and Explicit Semantic Re-ranking for Zero-Shot Composed Image Retrieval](https://mlsanity.com/paper/2604.14710/) (2026)
- [STiTch: Semantic Transition and Transportation in Collaboration for Training-Free Zero-Shot Composed Image Retrieval](https://mlsanity.com/paper/2605.21261/) (2026)

## Abstract (arXiv)

We argue that there are many notions of 'similarity' and that models, like humans, should be able to adapt to these dynamically. This contrasts with most representation learning methods, supervised or self-supervised, which learn a fixed embedding function and hence implicitly assume a single notion of similarity. For instance, models trained on ImageNet are biased towards object categories, while a user might prefer the model to focus on colors, textures or specific elements in the scene. In this paper, we propose the GeneCIS ('genesis') benchmark, which measures models' ability to adapt to a range of similarity conditions. Extending prior work, our benchmark is designed for zero-shot evaluation only, and hence considers an open-set of similarity conditions. We find that baselines from powerful CLIP models struggle on GeneCIS and that performance on the benchmark is only weakly correlated with ImageNet accuracy, suggesting that simply scaling existing methods is not fruitful. We further propose a simple, scalable solution based on automatically mining information from existing image-caption datasets. We find our method offers a substantial boost over the baselines on GeneCIS, and further improves zero-shot performance on related image retrieval benchmarks. In fact, though evaluated zero-shot, our model surpasses state-of-the-art supervised models on MIT-States. Project page at https://sgvaze.github.io/genecis/.