mlsanityData to 14 Oct 2026

TIGeR: Unifying Text-to-Image Generation and Retrieval with Large Multimodal Models

arXiv 2406.05814 ↗First posted 9 Jun 2024cs.CV cs.AI cs.CLcited by 12

On the map: Multimodal LLMs / vision-language models → Document understanding and multimodal RAG

Abstract (arXiv)

How humans can effectively and efficiently acquire images has always been a perennial question. A classic solution is text-to-image retrieval from an existing database; however, the limited database typically lacks creativity. By contrast, recent breakthroughs in text-to-image generation have made it possible to produce attractive and counterfactual visual content, but it faces challenges in synthesizing knowledge-intensive images. In this work, we rethink the relationship between text-to-image generation and retrieval, proposing a unified framework for both tasks with one single Large Multimodal Model (LMM). Specifically, we first explore the intrinsic discriminative abilities of LMMs and introduce an efficient generative retrieval method for text-to-image retrieval in a training-free manner. Subsequently, we unify generation and retrieval autoregressively and propose an autonomous decision mechanism to choose the best-matched one between generated and retrieved images as the response to the text prompt. To standardize the evaluation of unified text-to-image generation and retrieval, we construct TIGeR-Bench, a benchmark spanning both creative and knowledge-intensive domains. Extensive experiments on TIGeR-Bench and two retrieval benchmarks, i.e., Flickr30K and MS-COCO, demonstrate the superiority of our proposed framework.

Key facts. Cited by 12 papers in the map (data through 14 Oct 2026). Placed in Document understanding and multimodal RAG, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 73 of its 89 references that are in the map. First posted on arXiv 9 Jun 2024.

Builds on

73 of 89 references.

Closest ancestors · 1

Same direction, newest first

  1. 2024Generative Cross-Modal Retrieval: Memorizing Images in Multimodal Language Models for Retrieval and Beyond16 Feb

Other references · 72

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2020Denoising Diffusion Probabilistic Modelscited by 21,076
  3. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  4. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  5. 2022Training language models to follow instructions with human feedbackcited by 15,971
  6. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  7. 2023Visual Instruction Tuningcited by 7,869
  8. 2020Scaling Laws for Neural Language Modelscited by 6,586
  9. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,585
  10. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,355
  11. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
  12. 2017Neural Discrete Representation Learningcited by 4,712
38 more
  1. 2023SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesiscited by 4,356
  2. 2021Zero-Shot Text-to-Image Generationcited by 4,288
  3. 2024Scaling Rectified Flow Transformers for High-Resolution Image Synthesiscited by 4,181
  4. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,152
  5. 2021GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Modelscited by 3,614
  6. 2020Taming Transformers for High-Resolution Image Synthesiscited by 3,253
  7. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,224
  8. 2014Sequence to Sequence Learning with Neural Networkscited by 2,986
  9. 2021CLIPScore: A Reference-free Evaluation Metric for Image Captioningcited by 2,424
  10. 2021Align before Fuse: Vision and Language Representation Learning with Momentum Distillationcited by 1,619
  11. 2022CoCa: Contrastive Captioners are Image-Text Foundation Modelscited by 1,274
  12. 2015A Diversity-Promoting Objective Function for Neural Conversation Modelscited by 1,077
  13. 2023ImageReward: Learning and Evaluating Human Preferences for Text-to-Image Generationcited by 1,045
  14. 2023Pick-a-Pic: An Open Dataset of User Preferences for Text-to-Image Generationcited by 977
  15. 2016Generative Adversarial Text to Image Synthesiscited by 735
  16. 2023Attend-and-Excite: Attention-Based Semantic Guidance for Text-to-Image Diffusion Modelscited by 702
  17. 2021CogView: Mastering Text-to-Image Generation via Transformerscited by 694
  18. 2023NExT-GPT: Any-to-Any Multimodal LLMcited by 615
  19. 2023Generative Multimodal Models are In-Context Learnerscited by 446
  20. 2023Aligning Text-to-Image Models using Human Feedbackcited by 422
  21. 2018Stacked Cross Attention for Image-Text Matchingcited by 416
  22. 2019Your Classifier is Secretly an Energy Based Model and You Should Treat it Like Onecited by 405
  23. 2022Training-Free Structured Diffusion Guidance for Compositional Text-to-Image Synthesiscited by 388
  24. 2016Visual Dialogcited by 377
  25. 2020Google Landmarks Dataset v2 -- A Large-Scale Benchmark for Instance-Level Recognition and Retrievalcited by 309
  26. 2023Your Diffusion Model is Secretly a Zero-Shot Classifiercited by 306
  27. 2023Unified-IO 2: Scaling Autoregressive Multimodal Models with Vision, Language, Audio, and Actioncited by 302
  28. 2023DreamLLM: Synergistic Multimodal Comprehension and Creationcited by 297
  29. 2021An Empirical Study of Training End-to-End Vision-and-Language Transformerscited by 271
  30. 2023Generating Images with Multimodal Language Modelscited by 260
  31. 2022Transformer Memory as a Differentiable Search Indexcited by 247
  32. 2017Beam Search Strategies for Neural Machine Translationcited by 232
  33. 2024Mastering Text-to-Image Diffusion: Recaptioning, Planning, and Generating with Multimodal LLMscited by 224
  34. 2022Re-Imagen: Retrieval-Augmented Text-to-Image Generatorcited by 210
  35. 2019Visual Semantic Reasoning for Image-Text Matchingcited by 185
  36. 2023Making LLaMA SEE and Draw with SEED Tokenizercited by 185
  37. 2023Scaling Autoregressive Multi-Modal Models: Pretraining and Instruction Tuningcited by 170
  38. 2023UniIR: Training and Benchmarking Universal Multimodal Information Retrieverscited by 166

50 most cited of 72 All 72 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 2; 2025: 7; 2026: 3.

12 papers.

  1. 2024Visual Prompting in Multimodal Large Language Models: A Surveycited by 54
  2. 2025A Survey of Multimodal Retrieval-Augmented Generationcited by 37
  3. 2024A Survey on Multimodal Benchmarks: In the Era of Large AI Modelscited by 27
  4. 2025A Comprehensive Survey on Composed Image Retrievalcited by 17
  5. 2025VINCIE: Unlocking In-context Image Editing from Videocited by 17
  6. 2025Open Multimodal Retrieval-Augmented Factual Image Generationcited by 6
  7. 2025TTOM: Test-Time Optimization and Memorization for Compositional Video Generationcited by 4
  8. 2026ReCALL: Recalibrating Capability Degradation for MLLM-based Composed Image Retrievalcited by 3
  9. 2026WISER: Wider Search, Deeper Thinking, and Adaptive Fusion for Training-Free Zero-Shot Composed Image Retrievalcited by 3
  10. 2025Cross-modal RAG: Sub-dimensional Text-to-Image Retrieval-Augmented Generationcited by 1
  11. 2025Category-level Text-to-Image Retrieval Improved: Bridging the Domain Gap with Diffusion Models and Vision Encoderscited by 0
  12. 2026SLQ: Bridging Modalities via Shared Latent Queries for Retrieval with Frozen MLLMscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2022MuRAG: Multimodal Retrieval-Augmented Generator for Open Question Answering over Images and Text
  2. 2025ImageRAG: Dynamic Image Retrieval for Reference-Guided Image Generation
  3. 2024KITTEN: A Knowledge-Intensive Evaluation of Image Generation on Visual Entities
  4. 2025Generative Giants, Retrieval Weaklings: Why do Multimodal Large Language Models Fail at Multimodal Retrieval?
  5. 2024MMGenBench: Fully Automatically Evaluating LMMs from the Text-to-Image Generation Perspective
  6. 2024MRAG-Bench: Vision-Centric Evaluation for Retrieval-Augmented Multimodal Models
  7. 2025ImageScope: Unifying Language-Guided Image Retrieval via Large Multimodal Model Collective Reasoning
  8. 2022Retrieval-Augmented Multimodal Language Modeling