mlsanityData to 7 Oct 2026

OneLLM: One Framework to Align All Modalities with Language

arXiv 2312.03700 ↗First posted 6 Dec 2023cs.CV cs.AI cs.CLcited by 185

On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning

Abstract (arXiv)

Multimodal large language models (MLLMs) have gained significant attention due to their strong multimodal understanding capability. However, existing works rely heavily on modality-specific encoders, which usually differ in architecture and are limited to common modalities. In this paper, we present OneLLM, an MLLM that aligns eight modalities to language using a unified framework. We achieve this through a unified multimodal encoder and a progressive multimodal alignment pipeline. In detail, we first train an image projection module to connect a vision encoder with LLM. Then, we build a universal projection module (UPM) by mixing multiple image projection modules and dynamic routing. Finally, we progressively align more modalities to LLM with the UPM. To fully leverage the potential of OneLLM in following instructions, we also curated a comprehensive multimodal instruction dataset, including 2M items from image, audio, video, point cloud, depth/normal map, IMU and fMRI brain activity. OneLLM is evaluated on 25 diverse benchmarks, encompassing tasks such as multimodal captioning, question answering and reasoning, where it delivers excellent performance. Code, data, model and online demo are available at https://github.com/csuhan/OneLLM

Key facts. Cited by 185 papers in the map (data through 7 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 88 of its 106 references that are in the map. First posted on arXiv 6 Dec 2023.

Builds on

88 of 106 references.

Closest ancestors · 8

Same direction, newest first

  1. 2023Improved Baselines with Visual Instruction Tuning5 Oct
  2. 2023MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilities4 Aug
  3. 2023SEED-Bench: Benchmarking Multimodal LLMs with Generative Comprehension30 Jul
  4. 2023MMBench: Is Your Multi-modal Model an All-around Player?12 Jul
  5. 2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Models23 Jun
  6. 2023OBELICS: An Open Web-Scale Filtered Dataset of Interleaved Image-Text Documents21 Jun
  7. 2023SpeechGPT: Empowering Large Language Models with Intrinsic Cross-Modal Conversational Abilities18 May
  8. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning11 May

Other references · 80

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2023GPT-4 Technical Reportcited by 18,132
  3. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  4. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  5. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  6. 2023Visual Instruction Tuningcited by 7,792
  7. 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,764
  8. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  9. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  10. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,131
  11. 2023Qwen Technical Reportcited by 3,273
  12. 2022Training Compute-Optimal Large Language Modelscited by 2,927
38 more
  1. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,689
  2. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,646
  3. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,498
  4. 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answeringcited by 1,881
  5. 2021Vision Transformers for Dense Predictioncited by 1,789
  6. 2019Towards VQA Models That Can Readcited by 1,696
  7. 2021Ego4D: Around the World in 3,000 Hours of Egocentric Videocited by 1,530
  8. 2022Objaverse: A Universe of Annotated 3D Objectscited by 1,522
  9. 2023Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understandingcited by 1,461
  10. 2015Microsoft COCO Captions: Data Collection and Evaluation Servercited by 1,447
  11. 2015Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Modelscited by 1,352
  12. 2021Frozen in Time: A Joint Video and Image Encoder for End-to-End Retrievalcited by 1,175
  13. 2023ImageBind: One Embedding Space To Bind Them Allcited by 1,162
  14. 2019OK-VQA: A Visual Question Answering Benchmark Requiring External Knowledgecited by 1,114
  15. 2021Perceiver: General Perception with Iterative Attentioncited by 1,089
  16. 2023mPLUG-Owl: Modularization Empowers Large Language Models with Multimodalitycited by 1,038
  17. 2018VizWiz Grand Challenge: Answering Visual Questions from Blind Peoplecited by 852
  18. 2023VideoChat: Chat-Centric Video Understandingcited by 851
  19. 2023LLaMA-Adapter: Efficient Fine-tuning of Language Models with Zero-init Attentioncited by 776
  20. 2021NExT-QA:Next Phase of Question-Answering to Explaining Temporal Actionscited by 735
  21. 2022Large-scale Contrastive Language-Audio Pretraining with Feature Fusion and Keyword-to-Caption Augmentationcited by 707
  22. 2022A-OKVQA: A Benchmark for Visual Question Answering using World Knowledgecited by 702
  23. 2021CLIP4Clip: An Empirical Study of CLIP for End to End Video Clip Retrievalcited by 645
  24. 2019Three scenarios for continual learningcited by 610
  25. 2023LLaMA-Adapter V2: Parameter-Efficient Visual Instruction Modelcited by 602
  26. 2021VideoCLIP: Contrastive Pre-training for Zero-shot Video-Text Understandingcited by 596
  27. 2022GIT: A Generative Image-to-text Transformer for Vision and Languagecited by 536
  28. 2021PointCLIP: Point Cloud Understanding by CLIPcited by 461
  29. 2018nocaps: novel object captioning at scalecited by 444
  30. 2022InternVideo: General Video Foundation Models via Generative and Discriminative Learningcited by 441
  31. 2019VATEX: A Large-Scale, High-Quality Multilingual Dataset for Video-and-Language Researchcited by 412
  32. 2020TextCaps: a Dataset for Image Captioning with Reading Comprehensioncited by 398
  33. 2020HERO: Hierarchical Encoder for Video+Language Omni-representation Pre-trainingcited by 362
  34. 2019Clotho: An Audio Captioning Datasetcited by 354
  35. 2021Omnidata: A Scalable Pipeline for Making Multi-Task Mid-Level Vision Datasets from 3D Scanscited by 336
  36. 2021AudioCLIP: Extending CLIP to Image, Text and Audiocited by 335
  37. 2023PandaGPT: One Model To Instruction-Follow Them Allcited by 330
  38. 2023PointLLM: Empowering Large Language Models to Understand Point Cloudscited by 326

50 most cited of 80 All 80 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 3; 2024: 59; 2025: 78; 2026: 45.

185 papers.

Sort:

By area:

  1. 2024VideoLLaMA 2: Advancing Spatial-Temporal Modeling and Audio Understanding in Video-LLMscited by 749
  2. 2025VideoLLaMA 3: Frontier Multimodal Foundation Models for Image and Video Understandingcited by 502
  3. 2024VLMEvalKit: An Open-Source Toolkit for Evaluating Large Multi-Modality Modelscited by 443
  4. 2023Video Understanding with Large Language Models: A Surveycited by 203
  5. 2024On-Device Language Models: A Comprehensive Reviewcited by 101
  6. 2025Lifelong Learning of Large Language Model based Agents: A Roadmapcited by 78
  7. 2023Audio-Visual LLM for Video Understandingcited by 76
  8. 2024User-LLM: Efficient LLM Contextualization with User Embeddingscited by 74
  9. 2024Multimodal Alignment and Fusion: A Surveycited by 73
  10. 2024Towards Synergistic, Generalized, and Efficient Dual-System for Robotic Manipulationcited by 71
  11. 2024CAD-Recode: Reverse Engineering CAD Code from Point Cloudscited by 63
  12. 2024A Comprehensive Survey of Foundation Models in Medicinecited by 56
38 more
  1. 2024CAT: Enhancing Multimodal Large Language Model to Answer Questions in Dynamic Audio-Visual Scenarioscited by 53
  2. 2024AVHBench: A Cross-Modal Hallucination Benchmark for Audio-Visual Large Language Modelscited by 50
  3. 2024LongVALE: Vision-Audio-Language-Event Benchmark Towards Time-Aware Omni-Modal Perception of Long Videoscited by 49
  4. 2025PACT: Pruning and Clustering-Based Token Reduction for Faster Visual Language Modelscited by 49
  5. 2024MiniGPT-3D: Efficiently Aligning 3D Point Clouds with Large Language Models using 2D Priorscited by 48
  6. 2024The Curse of Multi-Modalities: Evaluating Hallucinations of Large Multimodal Models across Language, Visual, and Audiocited by 48
  7. 2024InternLM-XComposer2.5-OmniLive: A Comprehensive Multimodal System for Long-term Streaming Video and Audio Interactionscited by 48
  8. 2024Motion-Agent: A Conversational Framework for Human Motion Generation with LLMscited by 46
  9. 2024AV-Odyssey Bench: Can Your Multimodal LLMs Really Understand Audio-Visual Information?cited by 46
  10. 2024A Comprehensive Survey of Large Language Models and Multimodal Large Language Models in Medicinecited by 45
  11. 2025Vision as a Dialect: Unifying Visual Understanding and Generation via Text-Aligned Representationscited by 44
  12. 2025AffectGPT: A New Dataset, Model, and Benchmark for Emotion Understanding with Multimodal Large Language Modelscited by 43
  13. 2025Audio-centric Video Understanding Benchmark without Text Shortcutcited by 33
  14. 2024UMBRAE: Unified Multimodal Brain Decodingcited by 31
  15. 2024View Selection for 3D Captioning via Diffusion Rankingcited by 31
  16. 2024How Far Are We From AGI: Are LLMs All We Need?cited by 31
  17. 2025Towards Universal Modal Tracking with Online Dense Temporal Token Learningcited by 30
  18. 2025Temporal Working Memory: Query-Guided Segment Refinement for Enhanced Multimodal Understandingcited by 28
  19. 2025MM-OR: A Large Multimodal Operating Room Dataset for Semantic Understanding of High-Intensity Surgical Environmentscited by 28
  20. 2024Safety Alignment for Vision Language Modelscited by 27
  21. 2024EMO-LLaMA: Enhancing Facial Emotion Understanding with Instruction Tuningcited by 27
  22. 2025AVCD: Mitigating Hallucinations in Audio-Visual Large Language Models through Contrastive Decodingcited by 26
  23. 2024RAP: Retrieval-Augmented Personalization for Multimodal Large Language Modelscited by 24
  24. 2024Speech Translation with Speech Foundation Models and Large Language Models: What is There and What is Missing?cited by 23
  25. 2024MER 2024: Semi-Supervised Learning, Noise Robustness, and Open-Vocabulary Multimodal Emotion Recognitioncited by 23
  26. 2024GeoX: Geometric Problem Solving Through Unified Formalized Vision-Language Pre-trainingcited by 23
  27. 2024Cross-Modal Safety Alignment: Is textual unlearning all you need?cited by 22
  28. 2024From Specific-MLLMs to Omni-MLLMs: A Survey on MLLMs Aligned with Multi-modalitiescited by 22
  29. 2025Toward Generalizable Forgery Detection and Reasoningcited by 20
  30. 2025AVTrustBench: Assessing and Enhancing Reliability and Robustness in Audio-Visual LLMscited by 19
  31. 2025NExT-OMNI: Towards Any-to-Any Omnimodal Foundation Models with Discrete Flow Matchingcited by 19
  32. 2025See, Hear, and Understand: Benchmarking Audiovisual Human Speech Understanding in Multimodal Large Language Modelscited by 19
  33. 2024IoT-LLM: a framework for enhancing Large Language Model reasoning from real-world sensor datacited by 18
  34. 2024CREMA: Generalizable and Efficient Video-Language Reasoning via Multimodal Modular Fusioncited by 17
  35. 20253UR-LLM: An End-to-End Multimodal Large Language Model for 3D Scene Understandingcited by 16
  36. 2025AV-Reasoner: Improving and Benchmarking Clue-Grounded Audio-Visual Counting for MLLMscited by 15
  37. 2024Vision Search Assistant: Empower Vision-Language Models as Multimodal Search Enginescited by 14
  38. 2025MindLLM: A Subject-Agnostic and Versatile Model for fMRI-to-Text Decodingcited by 14

50 most cited of 185 All 185 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2023How to Bridge the Gap between Modalities: Survey on Multimodal Large Language Model
  2. 2024AnyGPT: Unified Multimodal LLM with Discrete Sequence Modeling
  3. 2024The Revolution of Multimodal Large Language Models: A Survey
  4. 2024EE-MLLM: A Data-Efficient and Compute-Efficient Multimodal Large Language Model
  5. 2024A Comprehensive Survey and Guide to Multimodal Large Language Models in Vision-Language Tasks
  6. 2025Aligning Multimodal LLM with Human Preference: A Survey
  7. 2023On the Performance of Multimodal Language Models
  8. 2026Strong Helps Weak: Directional Cross-Modal Alignment Transfer in Multi-modal LLMs