mlsanityData to 7 Oct 2026

ImageBind: One Embedding Space To Bind Them All

arXiv 2305.05665 ↗First posted 9 May 2023cs.CV cs.AI cs.LGcited by 1,162

Not on the map. Why?

Cited by 164 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

We present ImageBind, an approach to learn a joint embedding across six different modalities - images, text, audio, depth, thermal, and IMU data. We show that all combinations of paired data are not necessary to train such a joint embedding, and only image-paired data is sufficient to bind the modalities together. ImageBind can leverage recent large scale vision-language models, and extends their zero-shot capabilities to new modalities just by using their natural pairing with images. It enables novel emergent applications 'out-of-the-box' including cross-modal retrieval, composing modalities with arithmetic, cross-modal detection and generation. The emergent capabilities improve with the strength of the image encoder and we set a new state-of-the-art on emergent zero-shot recognition tasks across modalities, outperforming specialist supervised models. Finally, we show strong few-shot recognition results outperforming prior work, and that ImageBind serves as a new way to evaluate vision models for visual and non-visual tasks.

Key facts. Cited by 1,162 papers in the map (data through 7 Oct 2026). Builds on 72 of its 87 references that are in the map. First posted on arXiv 9 May 2023.

Builds on

72 of 87 references.

References · 72

  1. 2017Attention Is All You Needcited by 59,932
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  4. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  5. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,441
  6. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  7. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,893
  8. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,539
  9. 2020Training data-efficient image transformers & distillation through attentioncited by 4,560
  10. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  11. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,131
  12. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
38 more
  1. 2017Quo Vadis, Action Recognition? A New Model and the Kinetics Datasetcited by 3,259
  2. 2019RandAugment: Practical automated data augmentation with a reduced search spacecited by 2,261
  3. 2017The Kinetics Human Action Video Datasetcited by 1,894
  4. 2018Unsupervised Feature Learning via Non-Parametric Instance-level Discriminationcited by 1,838
  5. 2021LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairscited by 1,544
  6. 2021Ego4D: Around the World in 3,000 Hours of Egocentric Videocited by 1,530
  7. 2019Towards Robust Monocular Depth Estimation: Mixing Datasets for Zero-shot Cross-dataset Transfercited by 1,484
  8. 2022VideoMAE: Masked Autoencoders are Data-Efficient Learners for Self-Supervised Video Pre-Trainingcited by 1,305
  9. 2019Contrastive Multiview Codingcited by 1,297
  10. 2022CoCa: Contrastive Captioners are Image-Text Foundation Modelscited by 1,265
  11. 2017Random Erasing Data Augmentationcited by 1,235
  12. 2021Frozen in Time: A Joint Video and Image Encoder for End-to-End Retrievalcited by 1,175
  13. 2022Reproducible scaling laws for contrastive language-image learningcited by 1,163
  14. 2016Deep Networks with Stochastic Depthcited by 973
  15. 2019HowTo100M: Learning a Text-Video Embedding by Watching Hundred Million Narrated Video Clipscited by 946
  16. 2021Open-vocabulary Object Detection via Vision and Language Knowledge Distillationcited by 903
  17. 2021Florence: A New Foundation Model for Computer Visioncited by 802
  18. 2022Language-driven Semantic Segmentationcited by 711
  19. 2021CLIP4Clip: An Empirical Study of CLIP for End to End Video Clip Retrievalcited by 645
  20. 2020VGGSound: A Large-scale Audio-Visual Datasetcited by 641
  21. 2021AST: Audio Spectrogram Transformercited by 634
  22. 2022Detecting Twenty-thousand Classes using Image-level Supervisioncited by 613
  23. 2018Exploring the Limits of Weakly Supervised Pretrainingcited by 604
  24. 2021LiT: Zero-Shot Transfer with Locked-image text Tuningcited by 572
  25. 2019End-to-End Learning of Visual Representations from Uncurated Instructional Videoscited by 506
  26. 2016Google's Multilingual Neural Machine Translation System: Enabling Zero-Shot Translationcited by 467
  27. 2021PointCLIP: Point Cloud Understanding by CLIPcited by 461
  28. 2017Look, Listen and Learncited by 452
  29. 2022Masked Autoencoders As Spatiotemporal Learnerscited by 428
  30. 2021Attention Bottlenecks for Multimodal Fusioncited by 361
  31. 2018Audio-Visual Scene Analysis with Self-Supervised Multisensory Featurescited by 335
  32. 2021AudioCLIP: Extending CLIP to Image, Text and Audiocited by 335
  33. 2022BEiT v2: Masked Image Modeling with Vector-Quantized Visual Tokenizerscited by 298
  34. 2022MultiMAE: Multi-modal Multi-task Masked Autoencoderscited by 287
  35. 2020Self-Supervised MultiModal Versatile Networkscited by 276
  36. 2017Unsupervised Machine Translation Using Monolingual Corpora Onlycited by 272
  37. 2022Masked Autoencoders that Listencited by 271
  38. 2021LLVIP: A Visible-infrared Paired Dataset for Low-light Visioncited by 229

50 most cited of 72 All 72 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 165; 2024: 351; 2025: 364; 2026: 282.

1,162 papers.

Sort:

By area:

  1. 2023Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyondcited by 2,179
  2. 2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Modelscited by 1,624
  3. 2023Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understandingcited by 1,461
  4. 2023Video-LLaVA: Learning United Visual Representation by Alignment Before Projectioncited by 1,385
  5. 2023A Survey on Multimodal Large Language Modelscited by 898
  6. 2023PixArt-α: Fast Training of Diffusion Transformer for Photorealistic Text-to-Image Synthesiscited by 854
  7. 2023LRM: Large Reconstruction Model for Single Image to 3Dcited by 834
  8. 2024VideoLLaMA 2: Advancing Spatial-Temporal Modeling and Audio Understanding in Video-LLMscited by 749
  9. 2023Unifying Large Language Models and Knowledge Graphs: A Roadmapcited by 663
  10. 2023Instruction Tuning for Large Language Models: A Surveycited by 617
  11. 2023NExT-GPT: Any-to-Any Multimodal LLMcited by 610
  12. 2023MovieChat: From Dense Token to Sparse Memory for Long Video Understandingcited by 536
38 more
  1. 2023MotionGPT: Human Motion as a Foreign Languagecited by 521
  2. 2024Panda-70M: Captioning 70M Videos with Multiple Cross-Modality Teacherscited by 389
  3. 2023LanguageBind: Extending Video-Language Pretraining to N-modality by Language-based Semantic Alignmentcited by 366
  4. 2024A Survey on Hallucination in Large Vision-Language Modelscited by 352
  5. 2023AudioLDM 2: Learning Holistic Audio Generation with Self-supervised Pretrainingcited by 343
  6. 2023PandaGPT: One Model To Instruction-Follow Them Allcited by 330
  7. 2023PointLLM: Empowering Large Language Models to Understand Point Cloudscited by 326
  8. 2023Visual Adversarial Examples Jailbreak Aligned Large Language Modelscited by 307
  9. 2023Unified-IO 2: Scaling Autoregressive Multimodal Models with Vision, Language, Audio, and Actioncited by 301
  10. 2023GPT4RoI: Instruction Tuning Large Language Model on Region-of-Interestcited by 300
  11. 2023DreamLLM: Synergistic Multimodal Comprehension and Creationcited by 296
  12. 2023OpenMask3D: Open-Vocabulary 3D Instance Segmentationcited by 288
  13. 2023SPHINX: The Joint Mixing of Weights, Tasks, and Visual Embeddings for Multi-modal Large Language Modelscited by 283
  14. 2024MM-LLMs: Recent Advances in MultiModal Large Language Modelscited by 277
  15. 2024A Survey on Vision-Language-Action Models for Embodied AIcited by 274
  16. 2023Driving with LLMs: Fusing Object-Level Vector Modality for Explainable Autonomous Drivingcited by 231
  17. 2024Diffusion Model-Based Image Editing: A Surveycited by 212
  18. 2023RoboGen: Towards Unleashing Infinite Data for Automated Robot Learning via Generative Simulationcited by 206
  19. 2023Video Understanding with Large Language Models: A Surveycited by 203
  20. 2023Point-Bind & Point-LLM: Aligning Point Cloud with Multi-modality for 3D Understanding, Generation, and Instruction Followingcited by 192
  21. 2023Survey of Vulnerabilities in Large Language Models Revealed by Adversarial Attackscited by 189
  22. 2023OneLLM: One Framework to Align All Modalities with Languagecited by 185
  23. 2024MMBench-Video: A Long-Form Multi-Shot Benchmark for Holistic Video Understandingcited by 180
  24. 2024Aligning Cyber Space with Physical World: A Comprehensive Survey on Embodied AIcited by 175
  25. 2023AnomalyGPT: Detecting Industrial Anomalies Using Large Vision-Language Modelscited by 169
  26. 2023Macaw-LLM: Multi-Modal Language Modeling with Image, Audio, Video, and Text Integrationcited by 168
  27. 2023UniIR: Training and Benchmarking Universal Multimodal Information Retrieverscited by 165
  28. 2024Are Language Models Actually Useful for Time Series Forecasting?cited by 163
  29. 2024Scaling Proprioceptive-Visual Learning with Heterogeneous Pre-trained Transformerscited by 145
  30. 2024MiraData: A Large-Scale Video Dataset with Long Durations and Structured Captionscited by 144
  31. 2023ImageBind-LLM: Multi-modality Instruction Tuningcited by 142
  32. 2023Meta-Transformer: A Unified Framework for Multimodal Learningcited by 136
  33. 2024MMAudio: Taming Multimodal Joint Training for High-Quality Video-to-Audio Synthesiscited by 134
  34. 2024ShapeLLM: Universal 3D Object Understanding for Embodied Interactioncited by 133
  35. 2024Exploring scalable medical image encoders beyond text supervisioncited by 129
  36. 2023BLIVA: A Simple Multimodal LLM for Better Handling of Text-Rich Visual Questionscited by 125
  37. 2023DatasetDM: Synthesizing Data with Perception Annotations Using Diffusion Modelscited by 122
  38. 20234M: Massively Multimodal Masked Modelingcited by 118

50 most cited of 1,162 All 1,162 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Interpreting the linear structure of vision-language model embedding spaces
  2. 2023Intra-Modal Proxy Learning for Zero-Shot Visual Categorization with CLIP
  3. 2025Perception Encoder: The best visual embeddings are not at the output of the network
  4. 2024Harnessing Frozen Unimodal Encoders for Flexible Multimodal Alignment
  5. 2025ABC: Achieving Better Control of Multimodal Embeddings using VLMs
  6. 2026Cross-Modal Redundancy and the Geometry of Vision-Language Embeddings
  7. 2024VISTA: Visualized Text Embedding For Universal Multi-Modal Retrieval
  8. 2024BaFTA: Backprop-Free Test-Time Adaptation For Zero-Shot Vision-Language Models