mlsanityData to 7 Oct 2026

Can MLLMs Perform Text-to-Image In-Context Learning?

arXiv 2402.01293 ↗First posted 2 Feb 2024cs.LG cs.CLcited by 13

On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning

Abstract (arXiv)

The evolution from Large Language Models (LLMs) to Multimodal Large Language Models (MLLMs) has spurred research into extending In-Context Learning (ICL) to its multimodal counterpart. Existing such studies have primarily concentrated on image-to-text ICL. However, the Text-to-Image ICL (T2I-ICL), with its unique characteristics and potential applications, remains underexplored. To address this gap, we formally define the task of T2I-ICL and present CoBSAT, the first T2I-ICL benchmark dataset, encompassing ten tasks. Utilizing our dataset to benchmark six state-of-the-art MLLMs, we uncover considerable difficulties MLLMs encounter in solving T2I-ICL. We identify the primary challenges as the inherent complexity of multimodality and image generation, and show that strategies such as fine-tuning and Chain-of-Thought prompting help to mitigate these difficulties, leading to notable improvements in performance. Our code and dataset are available at https://github.com/UW-Madison-Lee-Lab/CoBSAT.

Key facts. Cited by 13 papers in the map (data through 7 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 86 of its 103 references that are in the map. First posted on arXiv 2 Feb 2024.

Builds on

86 of 103 references.

Closest ancestors · 8

Same direction, newest first

  1. 2023Can Multimodal Large Language Models Truly Perform Multimodal In-Context Learning?29 Nov
  2. 2023Improved Baselines with Visual Instruction Tuning5 Oct
  3. 2023DreamLLM: Synergistic Multimodal Comprehension and Creation20 Sep
  4. 2023MMICL: Empowering Vision-language Model with Multi-Modal In-Context Learning14 Sep
  5. 2023Scaling Autoregressive Multi-Modal Models: Pretraining and Instruction Tuning5 Sep
  6. 2023OBELICS: An Open Web-Scale Filtered Dataset of Interleaved Image-Text Documents21 Jun
  7. 2023Generating Images with Multimodal Language Models26 May
  8. 2023SpeechGPT: Empowering Large Language Models with Intrinsic Cross-Modal Conversational Abilities18 May

Other references · 78

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  4. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,604
  5. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  6. 2023GPT-4 Technical Reportcited by 18,132
  7. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  8. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  9. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  10. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  11. 2023Visual Instruction Tuningcited by 7,792
  12. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,005
38 more
  1. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,539
  2. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,601
  3. 2017Neural Discrete Representation Learningcited by 4,650
  4. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  5. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,310
  6. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,131
  7. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  8. 2022DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generationcited by 3,286
  9. 2023Qwen Technical Reportcited by 3,273
  10. 2023Tree of Thoughts: Deliberate Problem Solving with Large Language Modelscited by 3,155
  11. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,689
  12. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,646
  13. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,498
  14. 2021CLIPScore: A Reference-free Evaluation Metric for Image Captioningcited by 2,388
  15. 2023Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyondcited by 2,179
  16. 2023PaLM-E: An Embodied Multimodal Language Modelcited by 2,037
  17. 2015Microsoft COCO Captions: Data Collection and Evaluation Servercited by 1,447
  18. 2022Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?cited by 1,413
  19. 2020DocVQA: A Dataset for VQA on Document Imagescited by 1,288
  20. 2021Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Conceptscited by 1,209
  21. 2021What Makes Good In-Context Examples for GPT-3?cited by 1,184
  22. 2023Kosmos-2: Grounding Multimodal Large Language Models to the Worldcited by 1,075
  23. 2021An Explanation of In-context Learning as Implicit Bayesian Inferencecited by 915
  24. 2015Generation and Comprehension of Unambiguous Object Descriptionscited by 897
  25. 2022Transformers learn in-context by gradient descentcited by 714
  26. 2022What Can Transformers Learn In-Context? A Case Study of Simple Function Classescited by 700
  27. 2022EVA: Exploring the Limits of Masked Visual Representation Learning at Scalecited by 685
  28. 2022What learning algorithm is in-context learning? Investigations with linear modelscited by 671
  29. 2021Multimodal Few-Shot Learning with Frozen Language Modelscited by 657
  30. 2021Learning To Retrieve Prompts for In-Context Learningcited by 641
  31. 2022AudioLM: a Language Modeling Approach to Audio Generationcited by 611
  32. 2023OpenFlamingo: An Open-Source Framework for Training Large Autoregressive Vision-Language Modelscited by 560
  33. 2023Language Is Not All You Need: Aligning Perception with Language Modelscited by 543
  34. 2021MetaICL: Learning to Learn In Contextcited by 450
  35. 2023TIFA: Accurate and Interpretable Text-to-Image Faithfulness Evaluation with Question Answeringcited by 416
  36. 2023MLAgentBench: Evaluating Language Agents on Machine Learning Experimentationcited by 339
  37. 2018DVQA: Understanding Data Visualizations via Question Answeringcited by 338
  38. 2023Trained Transformers Learn Linear Models In-Contextcited by 331

50 most cited of 78 All 78 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 6; 2025: 6; 2026: 1.

13 papers.

Sort:

By area:

  1. 2025ImageGen-CoT: Enhancing Text-to-Image In-context Learning with Chain-of-Thought Reasoningcited by 37
  2. 2024VL-ICL Bench: The Devil in the Details of Multimodal In-Context Learningcited by 34
  3. 2024Next Token Prediction Towards Multimodal Intelligence: A Comprehensive Surveycited by 27
  4. 2024Can Large Vision-Language Models Correct Semantic Grounding Errors By Themselves?cited by 20
  5. 2024LoRA.rar: Learning to Merge LoRAs via Hypernetworks for Subject-Style Conditioned Image Generationcited by 19
  6. 2024Everything Everywhere All at Once: LLMs can In-Context Learn Multiple Tasks in Superpositioncited by 13
  7. 2025Unified Multimodal Understanding via Byte-Pair Visual Encodingcited by 9
  8. 2025In-Context Brush: Zero-shot Customized Subject Insertion with Context-Aware Latent Space Manipulationcited by 7
  9. 2025MemeSense: An Adaptive In-Context Framework for Social Commonsense Driven Meme Moderationcited by 3
  10. 2025CORDIAL: Can Multimodal Large Language Models Effectively Understand Coherence Relationships?cited by 3
  11. 2025POET: Preference Optimization for Enhanced Text-to-Image Generationcited by 3
  12. 2024ANCHOR: LLM-driven Subject Conditioning for Text-to-Image Synthesiscited by 0
1 more
  1. 2026Tree-of-Thoughts Reasoning for Text-to-Image In-Context Learningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024What Makes Multimodal In-Context Learning Work?
  2. 2024AIM: Let Any Multi-modal Large Language Models Embrace Efficient In-Context Learning
  3. 2025Boosting Text-To-Image Generation via Multilingual Prompting in Large Multimodal Models
  4. 2025What do vision-language models see in the context? Investigating multimodal in-context learning
  5. 2024Towards Multimodal In-Context Learning for Vision & Language Models
  6. 2024Evaluating Linguistic Capabilities of Multimodal LLMs in the Lens of Few-Shot Learning
  7. 2024The Revolution of Multimodal Large Language Models: A Survey
  8. 2024MileBench: Benchmarking MLLMs in Long Context