mlsanityData to 14 Oct 2026

ImageBind-LLM: Multi-modality Instruction Tuning

arXiv 2309.03905 ↗First posted 7 Sep 2023cs.MM cs.CL cs.CVcited by 142

Not on the map. Why?

Cited by 52 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

We present ImageBind-LLM, a multi-modality instruction tuning method of large language models (LLMs) via ImageBind. Existing works mainly focus on language and image instruction tuning, different from which, our ImageBind-LLM can respond to multi-modality conditions, including audio, 3D point clouds, video, and their embedding-space arithmetic by only image-text alignment training. During training, we adopt a learnable bind network to align the embedding space between LLaMA and ImageBind's image encoder. Then, the image features transformed by the bind network are added to word tokens of all layers in LLaMA, which progressively injects visual instructions via an attention-free and zero-initialized gating mechanism. Aided by the joint embedding of ImageBind, the simple image-text training enables our model to exhibit superior multi-modality instruction-following capabilities. During inference, the multi-modality inputs are fed into the corresponding ImageBind encoders, and processed by a proposed visual cache model for further cross-modal embedding enhancement. The training-free cache model retrieves from three million image features extracted by ImageBind, which effectively mitigates the training-inference modality discrepancy. Notably, with our approach, ImageBind-LLM can respond to instructions of diverse modalities and demonstrate significant language generation quality. Code is released at https://github.com/OpenGVLab/LLaMA-Adapter.

Key facts. Cited by 142 papers in the map (data through 14 Oct 2026). Builds on 61 of its 81 references that are in the map. First posted on arXiv 7 Sep 2023.

Builds on

61 of 81 references.

References · 61

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  3. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  4. 2023GPT-4 Technical Reportcited by 18,396
  5. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  6. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  7. 2023Visual Instruction Tuningcited by 7,869
  8. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
  9. 2016Layer Normalizationcited by 5,669
  10. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,386
  11. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,322
  12. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,152
38 more
  1. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,864
  2. 2016Gaussian Error Linear Units (GELUs)cited by 3,502
  3. 2023Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detectioncited by 3,332
  4. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,876
  5. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,711
  6. 2017Billion-scale similarity search with GPUscited by 2,497
  7. 2016Matching Networks for One Shot Learningcited by 2,255
  8. 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answeringcited by 1,919
  9. 2019Towards VQA Models That Can Readcited by 1,718
  10. 2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Modelscited by 1,652
  11. 2019Root Mean Square Layer Normalizationcited by 1,480
  12. 2023Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understandingcited by 1,471
  13. 2015Microsoft COCO Captions: Data Collection and Evaluation Servercited by 1,462
  14. 2020DocVQA: A Dataset for VQA on Document Imagescited by 1,317
  15. 2021BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-modelscited by 1,245
  16. 2021Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Conceptscited by 1,216
  17. 2023ImageBind: One Embedding Space To Bind Them Allcited by 1,175
  18. 2019OK-VQA: A Visual Question Answering Benchmark Requiring External Knowledgecited by 1,121
  19. 2021Florence: A New Foundation Model for Computer Visioncited by 804
  20. 2019Generalization through Memorization: Nearest Neighbor Language Modelscited by 800
  21. 2023LLaMA-Adapter: Efficient Fine-tuning of Language Models with Zero-init Attentioncited by 781
  22. 2023Instruction Tuning with GPT-4cited by 686
  23. 2023LLaMA-Adapter V2: Parameter-Efficient Visual Instruction Modelcited by 602
  24. 2023Otter: A Multi-Modal Model with In-Context Instruction Tuningcited by 550
  25. 2021PointCLIP: Point Cloud Understanding by CLIPcited by 463
  26. 2021Tip-Adapter: Training-free CLIP-Adapter for Better Vision-Language Modelingcited by 431
  27. 2016Visual Dialogcited by 377
  28. 2021AudioCLIP: Extending CLIP to Image, Text and Audiocited by 339
  29. 2023PandaGPT: One Model To Instruction-Follow Them Allcited by 334
  30. 2023Make-An-Audio: Text-To-Audio Generation with Prompt-Enhanced Diffusion Modelscited by 324
  31. 2022Visual Spatial Reasoningcited by 293
  32. 2021IconQA: A New Benchmark for Abstract Diagram Understanding and Visual Language Reasoningcited by 267
  33. 2021ICDAR2019 Competition on Scanned Receipt OCR and Information Extractioncited by 256
  34. 2021CLIP-Forge: Towards Zero-Shot Text-to-Shape Generationcited by 254
  35. 2023GPT4Tools: Teaching Large Language Model to Use Tools via Self-instructioncited by 248
  36. 2016COCO-Text: Dataset and Benchmark for Text Detection and Recognition in Natural Imagescited by 236
  37. 2023Multimodal C4: An Open, Billion-scale Corpus of Images Interleaved with Textcited by 208
  38. 2023Point-Bind & Point-LLM: Aligning Point Cloud with Multi-modality for 3D Understanding, Generation, and Instruction Followingcited by 194

50 most cited of 61 All 61 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 23; 2024: 62; 2025: 38; 2026: 19.

142 papers.

Sort:

By area:

  1. 2023Video-LLaVA: Learning United Visual Representation by Alignment Before Projectioncited by 1,402
  2. 2024MathVerse: Does Your Multi-modal LLM Truly See the Diagrams in Visual Math Problems?cited by 586
  3. 2024LLaVA-NeXT-Interleave: Tackling Multi-image, Video, and 3D in Large Multimodal Modelscited by 584
  4. 2024Hallucination of Multimodal Large Language Models: A Surveycited by 388
  5. 2024A Survey on Hallucination in Large Vision-Language Modelscited by 357
  6. 2023Unified-IO 2: Scaling Autoregressive Multimodal Models with Vision, Language, Audio, and Actioncited by 302
  7. 2023SPHINX: The Joint Mixing of Weights, Tasks, and Visual Embeddings for Multi-modal Large Language Modelscited by 286
  8. 2024VideoLLM-online: Online Video Large Language Model for Streaming Videocited by 225
  9. 2023OneLLM: One Framework to Align All Modalities with Languagecited by 187
  10. 2024Audio Flamingo: A Novel Audio Language Model with Few-Shot Learning and Dialogue Abilitiescited by 186
  11. 2023Chat-Scene: Bridging 3D Scene and Large Language Models with Object Identifierscited by 175
  12. 2024SPHINX-X: Scaling Data and Parameters for a Family of Multi-modal Large Language Modelscited by 131
38 more
  1. 2025Can We Generate Images with CoT? Let's Verify and Reinforce Image Generation Step by Stepcited by 131
  2. 2023Reason2Drive: Towards Interpretable and Chain-based Reasoning for Autonomous Drivingcited by 123
  3. 2024Grounded 3D-LLM with Referent Tokenscited by 115
  4. 2024MAVIS: Mathematical Visual Instruction Tuning with an Automatic Data Enginecited by 113
  5. 2024EmbSpatial-Bench: Benchmarking Spatial Understanding for Embodied Tasks with Large Vision-Language Modelscited by 107
  6. 2023Dynamic-SUPERB: Towards A Dynamic, Collaborative, and Comprehensive Instruction-Tuning Benchmark for Speechcited by 101
  7. 2024MMSearch: Benchmarking the Potential of Large Models as Multi-modal Search Enginescited by 101
  8. 2024Binding Touch to Everything: Learning Unified Multimodal Tactile Representationscited by 95
  9. 2023X-InstructBLIP: A Framework for aligning X-Modal instruction-aware representations to LLMs and Emergent Cross-modal Reasoningcited by 80
  10. 2024In-Context Imitation Learning via Next-Token Predictioncited by 70
  11. 2024BRAVE: Broadening the visual encoding of vision-language modelscited by 69
  12. 2024AVHBench: A Cross-Modal Hallucination Benchmark for Audio-Visual Large Language Modelscited by 56
  13. 2024CAT: Enhancing Multimodal Large Language Model to Answer Questions in Dynamic Audio-Visual Scenarioscited by 53
  14. 2025Multi-SpatialMLLM: Multi-Frame Spatial Understanding with Multi-Modal Large Language Modelscited by 52
  15. 2025VLAS: Vision-Language-Action Model With Speech Instructions For Customized Robot Manipulationcited by 51
  16. 2025Inst3D-LMM: Instance-Aware 3D Scene Understanding with Multi-modal Instruction Tuningcited by 51
  17. 2023A Challenger to GPT-4V? Early Explorations of Gemini in Visual Expertisecited by 49
  18. 2024ScanReason: Empowering 3D Visual Grounding with Reasoning Capabilitiescited by 47
  19. 2024MMWorld: Towards Multi-discipline Multi-faceted World Model Evaluation in Videoscited by 43
  20. 2024WorldGPT: Empowering LLM as Multimodal World Modelcited by 42
  21. 2024Harnessing the Power of MLLMs for Transferable Text-to-Image Person ReIDcited by 39
  22. 2024OmniBind: Large-scale Omni Multimodal Representation via Binding Spacescited by 39
  23. 2024From Redundancy to Relevance: Information Flow in LVLMs Across Reasoning Taskscited by 38
  24. 2024Large Language Models for Education: A Surveycited by 34
  25. 2024DeSTA: Enhancing Speech Language Models through Descriptive Speech-Text Alignmentcited by 33
  26. 20243DMIT: 3D Multi-modal Instruction Tuning for Scene Understandingcited by 32
  27. 2024How Far Are We From AGI: Are LLMs All We Need?cited by 31
  28. 2024Learning Modality-agnostic Representation for Semantic Segmentation from Any Modalitiescited by 31
  29. 2024PixWizard: Versatile Image-to-Image Visual Assistant with Open-Language Instructionscited by 31
  30. 2024Where Visual Speech Meets Language: VSP-LLM Framework for Efficient and Context-Aware Visual Speech Processingcited by 30
  31. 2023LLark: A Multimodal Instruction-Following Language Model for Musiccited by 29
  32. 2023ViT-Lens: Towards Omni-modal Representationscited by 29
  33. 2023Visual Instruction Tuning towards General-Purpose Multimodal Model: A Surveycited by 29
  34. 2023LEGO: Learning EGOcentric Action Frame Generation via Visual Instruction Tuningcited by 28
  35. 2024Robin3D: Improving 3D Large Language Model via Robust Instruction Tuningcited by 26
  36. 2024RAG and RAU: A Survey on Retrieval-Augmented Language Model in Natural Language Processingcited by 25
  37. 2023Improving Compositional Text-to-image Generation with Large Vision-Language Modelscited by 24
  38. 2024Benchmarking Large Multimodal Models against Common Corruptionscited by 24

50 most cited of 142 All 142 (JSON)