mlsanityData to 7 Oct 2026

LanguageBind: Extending Video-Language Pretraining to N-modality by Language-based Semantic Alignment

arXiv 2310.01852 ↗First posted 3 Oct 2023cs.CV cs.AIcited by 366

Not on the map. Why?

Cited by 94 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

The video-language (VL) pretraining has achieved remarkable improvement in multiple downstream tasks. However, the current VL pretraining framework is hard to extend to multiple modalities (N modalities, N>=3) beyond vision and language. We thus propose LanguageBind, taking the language as the bind across different modalities because the language modality is well-explored and contains rich semantics. Specifically, we freeze the language encoder acquired by VL pretraining, then train encoders for other modalities with contrastive learning. As a result, all modalities are mapped to a shared feature space, implementing multi-modal semantic alignment. While LanguageBind ensures that we can extend VL modalities to N modalities, we also need a high-quality dataset with alignment data pairs centered on language. We thus propose VIDAL-10M with Video, Infrared, Depth, Audio and their corresponding Language, naming as VIDAL-10M. In our VIDAL-10M, all videos are from short video platforms with complete semantics rather than truncated segments from long videos, and all the video, depth, infrared, and audio modalities are aligned to their textual descriptions. LanguageBind has achieved superior performance on a wide range of 15 benchmarks covering video, audio, depth, and infrared. Moreover, multiple experiments have provided evidence for the effectiveness of LanguageBind in achieving indirect alignment and complementarity among diverse modalities. Code address: https://github.com/PKU-YuanGroup/LanguageBind

Key facts. Cited by 366 papers in the map (data through 7 Oct 2026). Builds on 43 of its 76 references that are in the map. First posted on arXiv 3 Oct 2023.

Builds on

43 of 76 references.

References · 43

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  4. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  5. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  6. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  7. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  8. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,339
  9. 2017The Kinetics Human Action Video Datasetcited by 1,894
  10. 2019Transformer-XL: Attentive Language Models Beyond a Fixed-Length Contextcited by 1,830
  11. 2021Ego4D: Around the World in 3,000 Hours of Egocentric Videocited by 1,530
  12. 2021Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Conceptscited by 1,209
31 more
  1. 2023ImageBind: One Embedding Space To Bind Them Allcited by 1,162
  2. 2014Two-Stream Convolutional Networks for Action Recognition in Videoscited by 1,067
  3. 2023mPLUG-Owl: Modularization Empowers Large Language Models with Multimodalitycited by 1,038
  4. 2019HowTo100M: Learning a Text-Video Embedding by Watching Hundred Million Narrated Video Clipscited by 946
  5. 2022Large-scale Contrastive Language-Audio Pretraining with Feature Fusion and Keyword-to-Caption Augmentationcited by 707
  6. 2017Localizing Moments in Video with Natural Languagecited by 660
  7. 2021CLIP4Clip: An Empirical Study of CLIP for End to End Video Clip Retrievalcited by 645
  8. 2020VGGSound: A Large-scale Audio-Visual Datasetcited by 641
  9. 2021PointCLIP: Point Cloud Understanding by CLIPcited by 461
  10. 2017AVA: A Video Dataset of Spatio-temporally Localized Atomic Visual Actionscited by 449
  11. 2022InternVideo: General Video Foundation Models via Generative and Discriminative Learningcited by 441
  12. 2016YouTube-8M: A Large-Scale Video Classification Benchmarkcited by 324
  13. 2022Scaling Language-Image Pre-training via Maskingcited by 317
  14. 2020ActBERT: Learning Global-Local Video-Text Representationscited by 267
  15. 2020UniVL: A Unified Video and Language Pre-Training Model for Multimodal Understanding and Generationcited by 258
  16. 2021LLVIP: A Visible-infrared Paired Dataset for Low-light Visioncited by 229
  17. 2021Advancing High-Resolution Video-Language Representation with Large-Scale Video Transcriptionscited by 226
  18. 2023Unmasked Teacher: Towards Training-Efficient Video Foundation Modelscited by 199
  19. 2023VAST: A Vision-Audio-Subtitle-Text Omni-Modality Foundation Model and Datasetcited by 168
  20. 2023AIM: Adapting Image Models for Efficient Video Action Recognitioncited by 145
  21. 2022OmniVL:One Foundation Model for Image-Language and Video-Language Taskscited by 143
  22. 2023VALOR: Vision-Audio-Language Omni-Perception Pretraining Model and Datasetcited by 138
  23. 2023Meta-Transformer: A Unified Framework for Multimodal Learningcited by 136
  24. 2018Charades-Ego: A Large-Scale Dataset of Paired Third and First Person Videoscited by 127
  25. 2016Estimating Depth from Monocular Images as Classification Using Deep Fully Convolutional Residual Networkscited by 91
  26. 2022Learning Audio-Video Modalities from Image Captionscited by 80
  27. 2021A Straightforward Framework For Video Retrieval Using CLIPcited by 66
  28. 2022Global-Local Path Networks for Monocular Depth Estimation with Vertical CutDepthcited by 58
  29. 2020Rethinking CNN Models for Audio Classificationcited by 48
  30. 2023Edge-guided Multi-domain RGB-to-TIR image Translation for Training Vision Tasks with Challenging Labelscited by 30
  31. 2023TVTSv2: Learning Out-of-the-box Spatiotemporal Visual Representations at Scalecited by 14

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 7; 2024: 120; 2025: 147; 2026: 92.

366 papers.

Sort:

By area:

  1. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Taskscited by 2,534
  2. 2024LlamaFactory: Unified Efficient Fine-Tuning of 100+ Language Modelscited by 1,442
  3. 2023Video-LLaVA: Learning United Visual Representation by Alignment Before Projectioncited by 1,385
  4. 2024VideoLLaMA 2: Advancing Spatial-Temporal Modeling and Audio Understanding in Video-LLMscited by 749
  5. 2024LLaVA-Video: Video Instruction Tuning With Synthetic Datacited by 550
  6. 2024SparseVLM: Visual Token Sparsification for Efficient Vision-Language Model Inferencecited by 378
  7. 2023Chat-UniVi: Unified Visual Representation Empowers Large Language Models with Image and Video Understandingcited by 368
  8. 2024InternVideo2: Scaling Foundation Models for Multimodal Video Understandingcited by 310
  9. 2024VisionZip: Longer is Better but Not Necessary in Vision Language Modelscited by 298
  10. 2024Open-Sora Plan: Open-Source Large Video Generation Modelcited by 286
  11. 2024MM-LLMs: Recent Advances in MultiModal Large Language Modelscited by 277
  12. 2024MLVU: Benchmarking Multi-task Long Video Understandingcited by 275
38 more
  1. 2024A Survey on Vision-Language-Action Models for Embodied AIcited by 274
  2. 2023Video Understanding with Large Language Models: A Surveycited by 203
  3. 2024Direct Preference Optimization of Video Large Multimodal Models from Language Model Rewardcited by 145
  4. 2024MMAudio: Taming Multimodal Joint Training for High-Quality Video-to-Audio Synthesiscited by 134
  5. 2025Advances and Challenges in Foundation Agents: From Brain-Inspired Intelligence to Evolutionary, Collaborative, and Safe Systemscited by 125
  6. 2024AdaShield: Safeguarding Multimodal Large Language Models from Structure-based Attack via Adaptive Shield Promptingcited by 113
  7. 2024VideoPrism: A Foundational Visual Encoder for Video Understandingcited by 103
  8. 2024TemporalBench: Benchmarking Fine-grained Temporal Understanding for Multimodal Video Modelscited by 96
  9. 2024Efficient Multimodal Large Language Models: A Surveycited by 95
  10. 2024RAG-Driver: Generalisable Driving Explanations with Retrieval-Augmented In-Context Learning in Multi-Modal Large Language Modelcited by 91
  11. 2024MotionLLM: Understanding Human Behaviors from Human Motions and Videoscited by 89
  12. 2024Deep Multimodal Learning with Missing Modality: A Surveycited by 89
  13. 2024Apollo: An Exploration of Video Understanding in Large Multimodal Modelscited by 87
  14. 2024A Survey of Resource-efficient LLM and Multimodal Foundation Modelscited by 84
  15. 2024WildVision: Evaluating Vision-Language Models in the Wild with Human Preferencescited by 76
  16. 2024TEOChat: A Large Vision-Language Assistant for Temporal Earth Observation Datacited by 69
  17. 2024TimeMarker: A Versatile Video-LLM for Long and Short Video Understanding with Superior Temporal Localization Abilitycited by 69
  18. 2024E.T. Bench: Towards Open-Ended Event-Level Video-Language Understandingcited by 68
  19. 2024VidHalluc: Evaluating Temporal Hallucinations in Multimodal Large Language Models for Video Understandingcited by 65
  20. 2025UniWorld-V1: High-Resolution Semantic Encoders for Unified Visual Understanding and Generationcited by 64
  21. 2024Holmes-VAD: Towards Unbiased and Explainable Video Anomaly Detection via Multi-modal LLMcited by 63
  22. 2025Towards Understanding Camera Motions in Any Videocited by 62
  23. 2024Visual-RolePlay: Universal Jailbreak Attack on MultiModal Large Language Models via Role-playing Image Charactercited by 61
  24. 2024WF-VAE: Enhancing Video VAE by Wavelet-Driven Energy Flow for Latent Video Diffusion Modelcited by 55
  25. 2024KeyVideoLLM: Towards Large-scale Video Keyframe Selectioncited by 52
  26. 2024Gramian Multimodal Representation Learning and Alignmentcited by 52
  27. 2024MagicTime: Time-lapse Video Generation Models as Metamorphic Simulatorscited by 51
  28. 2024ReXTime: A Benchmark Suite for Reasoning-Across-Time in Videoscited by 51
  29. 2025STORM: Token-Efficient Long Video Understanding for Multimodal LLMscited by 46
  30. 2024OmniFlow: Any-to-Any Generation with Multi-Modal Rectified Flowscited by 45
  31. 2024When Large Language Model Agents Meet 6G Networks: Perception, Grounding, and Alignmentcited by 43
  32. 2024AIM: Adaptive Inference of Multi-Modal LLMs via Token Merging and Pruningcited by 43
  33. 2024WorldGPT: Empowering LLM as Multimodal World Modelcited by 42
  34. 2024Sports-QA: A Large-Scale Video Question Answering Benchmark for Complex and Professional Sportscited by 41
  35. 2024UNIAA: A Unified Multi-modal Image Aesthetic Assessment Baseline and Benchmarkcited by 39
  36. 2024OmniBind: Large-scale Omni Multimodal Representation via Binding Spacescited by 38
  37. 2025MLLMs are Deeply Affected by Modality Biascited by 38
  38. 2024Vinoground: Scrutinizing LMMs over Dense Temporal Reasoning with Short Videoscited by 37

50 most cited of 366 All 366 (JSON)