mlsanityData to 14 Oct 2026

TextBind: Multi-turn Interleaved Multimodal Instruction-following in the Wild

arXiv 2309.08637 ↗First posted 14 Sep 2023cs.CL cs.AIcited by 22

Not on the map. Why?

Abstract (arXiv)

Large language models with instruction-following abilities have revolutionized the field of artificial intelligence. These models show exceptional generalizability to tackle various real-world tasks through their natural language interfaces. However, their performance heavily relies on high-quality exemplar data, which is often difficult to obtain. This challenge is further exacerbated when it comes to multimodal instruction following. We introduce TextBind, an almost annotation-free framework for empowering larger language models with the multi-turn interleaved multimodal instruction-following capabilities. Our approach requires only image-caption pairs and generates multi-turn multimodal instruction-response conversations from a language model. To accommodate interleaved image-text inputs and outputs, we devise MIM, a language model-centric architecture that seamlessly integrates image encoder and decoder models. We release our dataset, model, and demo to foster future research in the area of multimodal instruction following.

Key facts. Cited by 22 papers in the map (data through 14 Oct 2026). Builds on 49 of its 72 references that are in the map. First posted on arXiv 14 Sep 2023.

Builds on

49 of 72 references.

References · 49

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  4. 2023GPT-4 Technical Reportcited by 18,396
  5. 2022Training language models to follow instructions with human feedbackcited by 15,971
  6. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  7. 2018The Unreasonable Effectiveness of Deep Features as a Perceptual Metriccited by 11,177
  8. 2023Visual Instruction Tuningcited by 7,869
  9. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
  10. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,512
  11. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,386
  12. 2023SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesiscited by 4,356
37 more
  1. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,322
  2. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,780
  3. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,876
  4. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,724
  5. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,711
  6. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,516
  7. 2017Billion-scale similarity search with GPUscited by 2,497
  8. 2023MMBench: Is Your Multi-modal Model an All-around Player?cited by 2,141
  9. 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answeringcited by 1,919
  10. 2019Towards VQA Models That Can Readcited by 1,718
  11. 2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Modelscited by 1,652
  12. 2021Multitask Prompted Training Enables Zero-Shot Task Generalizationcited by 1,470
  13. 2015Microsoft COCO Captions: Data Collection and Evaluation Servercited by 1,462
  14. 2023MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilitiescited by 1,239
  15. 2021Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Conceptscited by 1,216
  16. 2023HuggingGPT: Solving AI Tasks with ChatGPT and its Friends in Hugging Facecited by 1,190
  17. 2019OK-VQA: A Visual Question Answering Benchmark Requiring External Knowledgecited by 1,121
  18. 2023mPLUG-Owl: Modularization Empowers Large Language Models with Multimodalitycited by 1,040
  19. 2021GLM: General Language Model Pretraining with Autoregressive Blank Infillingcited by 1,031
  20. 2023Shikra: Unleashing Multimodal LLM's Referential Dialogue Magiccited by 906
  21. 2018VizWiz Grand Challenge: Answering Visual Questions from Blind Peoplecited by 858
  22. 2023LLaMA-Adapter: Efficient Fine-tuning of Language Models with Zero-init Attentioncited by 781
  23. 2023Visual ChatGPT: Talking, Drawing and Editing with Visual Foundation Modelscited by 668
  24. 2017Towards Automatic Learning of Procedures from Web Instructional Videoscited by 619
  25. 2023ViperGPT: Visual Inference via Python Execution for Reasoningcited by 603
  26. 2023Otter: A Multi-Modal Model with In-Context Instruction Tuningcited by 550
  27. 2023MM-REACT: Prompting ChatGPT for Multimodal Reasoning and Actioncited by 517
  28. 2023Mitigating Hallucination in Large Multi-Modal Models via Robust Instruction Tuningcited by 445
  29. 2018nocaps: novel object captioning at scalecited by 444
  30. 2022Generalized Decoding for Pixel, Image, and Languagecited by 294
  31. 2023Generating Images with Multimodal Language Modelscited by 260
  32. 2022A Contrastive Framework for Neural Text Generationcited by 225
  33. 2023LVLM-eHub: A Comprehensive Evaluation Benchmark for Large Vision-Language Modelscited by 217
  34. 2023Grounding Language Models to Images for Multimodal Inputs and Outputscited by 131
  35. 2023M3IT: A Large-Scale Dataset towards Multi-Modal Multilingual Instruction Tuningcited by 126
  36. 2022MultiInstruct: Improving Multi-Modal Zero-Shot Learning via Instruction Tuningcited by 109
  37. 2021Multimodal Dialogue Response Generationcited by 32

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 4; 2024: 7; 2025: 7; 2026: 4.

22 papers.

Sort:

By area:

  1. 2024The Revolution of Multimodal Large Language Models: A Surveycited by 137
  2. 2024Multimodal Alignment and Fusion: A Surveycited by 74
  3. 2023Igniting Language Intelligence: The Hitchhiker's Guide From Chain-of-Thought Reasoning to Language Agentscited by 72
  4. 2025MM-IFEngine: Towards Multimodal Instruction Followingcited by 42
  5. 2023GPT4Video: A Unified Multimodal Large Language Model for lnstruction-Followed Understanding and Safety-Aware Generationcited by 36
  6. 2023Visual Instruction Tuning towards General-Purpose Multimodal Model: A Surveycited by 29
  7. 2024LLaMo: Large Language Model-based Molecular Graph Assistantcited by 22
  8. 2025M2-omni: Advancing Omni-MLLM for Comprehensive Modality Support with Competitive Performancecited by 19
  9. 2025Multi-turn Consistent Image Editingcited by 18
  10. 2023Vision-Language Instruction Tuning: A Review and Analysiscited by 17
  11. 2025MMCR: Advancing Visual Language Model in Multimodal Multi-Turn Contextual Reasoningcited by 16
  12. 2025SynthRL: Scaling Visual Reasoning with Verifiable Data Synthesiscited by 12
10 more
  1. 2026UniM: A Unified Any-to-Any Interleaved Multimodal Benchmarkcited by 11
  2. 2024IDA-VLM: Towards Movie Understanding via ID-Aware Large Vision-Language Modelcited by 10
  3. 2024Modality-Specialized Synergizers for Interleaved Vision-Language Generalistscited by 9
  4. 2024Multi-modal Retrieval Augmented Multi-modal Generation: Datasets, Evaluation Metrics and Strong Baselinescited by 9
  5. 2024MM-Instruct: Generated Visual Instructions for Large Multimodal Model Alignmentcited by 5
  6. 2026Unveiling Fine-Grained Visual Traces: Evaluating Multimodal Interleaved Reasoning Chains in Multimodal STEM Taskscited by 4
  7. 2025Memory Helps, but Confabulation Misleads: Understanding Streaming Events in Videos with MLLMscited by 3
  8. 2025ViSTA: Visual Storytelling using Multi-modal Adapters for Text-to-Image Diffusion Modelscited by 3
  9. 2026DeDPO: Debiased Direct Preference Optimization for Diffusion Modelscited by 1
  10. 2026Towards Scalable RLVR: Multimodal Instruction Following Data Synthesis and Distillationcited by 0