mlsanityData to 14 Oct 2026

Category-level Text-to-Image Retrieval Improved: Bridging the Domain Gap with Diffusion Models and Vision Encoders

arXiv 2509.00177 ↗First posted 29 Aug 2025cs.CVcited by 0

On the map: Diffusion models → Image generation, text-to-image and editing

Abstract (arXiv)

This work explores text-to-image retrieval for queries that specify or describe a semantic category. While vision-and-language models (VLMs) like CLIP offer a straightforward open-vocabulary solution, they map text and images to distant regions in the representation space, limiting retrieval performance. To bridge this modality gap, we propose a two-step approach. First, we transform the text query into a visual query using a generative diffusion model. Then, we estimate image-to-image similarity with a vision model. Additionally, we introduce an aggregation network that combines multiple generated images into a single vector representation and fuses similarity scores across both query modalities. Our approach leverages advancements in vision encoders, VLMs, and text-to-image generation models. Extensive evaluations show that it consistently outperforms retrieval methods relying solely on text queries. Source code is available at: https://github.com/faixan-khan/cletir

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Image generation, text-to-image and editing, one of 14 directions in Diffusion models. Builds on 58 of its 79 references that are in the map. First posted on arXiv 29 Aug 2025.

Builds on

58 of 79 references.

Closest ancestors · 5

Same direction, newest first

  1. 2024Diffusion Feedback Helps CLIP See Better29 Jul
  2. 2023DataComp: In search of the next generation of multimodal datasets27 Apr
  3. 2023Synthetic Data from Diffusion Models Improves ImageNet Classification17 Apr
  4. 2022Fake it till you make it: Learning transferable representations from synthetic ImageNet clones16 Dec
  5. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding23 May

Other references · 53

  1. 2017Attention Is All You Needcited by 60,361
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  4. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  5. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,498
  6. 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,974
  7. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,968
  8. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,322
  9. 2021Zero-Shot Text-to-Image Generationcited by 4,288
  10. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,152
  11. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,864
  12. 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,109
38 more
  1. 2019Benchmarking Neural Network Robustness to Common Corruptions and Perturbationscited by 2,877
  2. 2013Describing Textures in the Wildcited by 2,125
  3. 2013Fine-Grained Visual Classification of Aircraftcited by 1,726
  4. 2022Conditional Prompt Learning for Vision-Language Modelscited by 1,433
  5. 2022CoCa: Contrastive Captioners are Image-Text Foundation Modelscited by 1,274
  6. 2022Reproducible scaling laws for contrastive language-image learningcited by 1,173
  7. 2024Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMscited by 838
  8. 2021Florence: A New Foundation Model for Computer Visioncited by 804
  9. 2017Remote Sensing Image Scene Classification: Benchmark and State of the Artcited by 797
  10. 2023Adversarial Diffusion Distillationcited by 759
  11. 2024Eyes Wide Shut? Exploring the Visual Shortcomings of Multimodal LLMscited by 735
  12. 2022EVA: Exploring the Limits of Masked Visual Representation Learning at Scalecited by 691
  13. 2022Mind the Gap: Understanding the Modality Gap in Multi-modal Contrastive Representation Learningcited by 561
  14. 2021Tip-Adapter: Training-free CLIP-Adapter for Better Vision-Language Modelingcited by 431
  15. 2024LongVU: Spatiotemporal Adaptive Compression for Long Video-Language Understandingcited by 356
  16. 2019A Short Note on the Kinetics-700 Human Action Datasetcited by 310
  17. 2019Visual Semantic Reasoning for Image-Text Matchingcited by 185
  18. 2023Prompt, Generate, then Cache: Cascade of Foundation Models makes Strong Few-shot Learnerscited by 152
  19. 2021Probabilistic Embeddings for Cross-Modal Retrievalcited by 151
  20. 2020Learning the Best Pooling Strategy for Visual Semantic Embeddingcited by 137
  21. 2022SuS-X: Training-Free Name-Only Transfer of Vision-Language Modelscited by 119
  22. 2023Dataset Diffusion: Diffusion-based Synthetic Dataset Generation for Pixel-Level Semantic Segmentationcited by 112
  23. 2022Hyperbolic Vision Transformers: Combining Improvements in Metric Learningcited by 111
  24. 2024ProxyCLIP: Proxy Attention Improves CLIP for Open-Vocabulary Segmentationcited by 111
  25. 2024EVA-CLIP-18B: Scaling CLIP to 18 Billion Parameterscited by 99
  26. 2023CLIP-DINOiser: Teaching CLIP a few DINO tricks for open-vocabulary semantic segmentationcited by 96
  27. 2019Polysemous Visual-Semantic Embedding for Cross-Modal Retrievalcited by 83
  28. 2021Deep Learning for Instance Retrieval: A Surveycited by 70
  29. 2017Learning Semantic Concepts and Order for Image and Sentence Matchingcited by 62
  30. 2023FreeMask: Synthetic Images with Dense Annotations Make Stronger Segmentation Modelscited by 61
  31. 2023Learning Customized Visual Models with Retrieval-Augmented Knowledgecited by 58
  32. 2023Frozen Transformers in Language Models Are Effective Visual Encoder Layerscited by 39
  33. 2024In Defense of Lazy Visual Grounding for Open-Vocabulary Semantic Segmentationcited by 38
  34. 2023Unicom: Universal and Compact Representation Learning for Image Retrievalcited by 36
  35. 2023Retrieval-Enhanced Contrastive Vision-Text Modelscited by 36
  36. 2024Label Propagation for Zero-shot Classification with Vision-Language Modelscited by 28
  37. 2024MoDE: CLIP Data Experts via Clusteringcited by 22
  38. 2025LPOSS: Label Propagation Over Patches and Pixels for Open-vocabulary Semantic Segmentationcited by 21

50 most cited of 53 All 53 (JSON)

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Vision-Free Retrieval: Rethinking Multimodal Search with Textual Scene Descriptions
  2. 2026Seeing Through Words: Controlling Visual Retrieval Quality with Language Models
  3. 2025A Little More Like This: Text-to-Image Retrieval with Vision-Language Models Using Relevance Feedback
  4. 2024Adapting Dual-encoder Vision-language Models for Paraphrased Retrieval
  5. 2024Rethinking Sparse Lexical Representations for Image Retrieval in the Age of Rising Multi-Modal Large Language Models
  6. 2025SERVAL: Surprisingly Effective Zero-Shot Visual Document Retrieval Powered by Large Vision and Language Models
  7. 2025Diffusion Augmented Retrieval: A Training-Free Approach to Interactive Text-to-Image Retrieval
  8. 2024VISTA: Visualized Text Embedding For Universal Multi-Modal Retrieval