mlsanityData to 7 Oct 2026

LLaVA-OneVision: Easy Visual Task Transfer

arXiv 2408.03326 ↗First posted 6 Aug 2024cs.CV cs.AI cs.CLcited by 2,669

On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning

Abstract (arXiv)

We present LLaVA-OneVision, a family of open large multimodal models (LMMs) developed by consolidating our insights into data, models, and visual representations in the LLaVA-NeXT blog series. Our experimental results demonstrate that LLaVA-OneVision is the first single model that can simultaneously push the performance boundaries of open LMMs in three important computer vision scenarios: single-image, multi-image, and video scenarios. Importantly, the design of LLaVA-OneVision allows strong transfer learning across different modalities/scenarios, yielding new emerging capabilities. In particular, strong video understanding and cross-scenario capabilities are demonstrated through task transfer from images to videos.

Key facts. Cited by 2,669 papers in the map (data through 7 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 119 of its 156 references that are in the map. First posted on arXiv 6 Aug 2024.

Builds on

119 of 156 references.

Closest ancestors · 21

Same direction, newest first

  1. 2024LMMs-Eval: Reality Check on the Evaluation of Large Multimodal Models17 Jul
  2. 2024Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMs24 Jun
  3. 2024MuirBench: A Comprehensive Benchmark for Robust Multi-image Understanding13 Jun
  4. 2024Vibe-Eval: A hard evaluation suite for measuring progress of multimodal language models3 May
  5. 2024MANTIS: Interleaved Multi-Image Instruction Tuning2 May
  6. 2024BLINK: Multimodal Large Language Models Can See but Not Perceive18 Apr
  7. 2024Direct Preference Optimization of Video Large Multimodal Models from Language Model Reward1 Apr
  8. 2024MM1: Methods, Analysis & Insights from Multimodal LLM Pre-training14 Mar
  9. 2024Vision-Flan: Scaling Human-Labeled Tasks in Visual Instruction Tuning18 Feb
  10. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Tasks21 Dec
  11. 2023VILA: On Pre-training for Visual Language Models12 Dec
  12. 2023ShareGPT4V: Improving Large Multi-Modal Models with Better Captions21 Nov
9 more
  1. 2023UReader: Universal OCR-free Visually-situated Language Understanding with Multimodal Large Language Model8 Oct
  2. 2023Improved Baselines with Visual Instruction Tuning5 Oct
  3. 2023Fine-tuning Multimodal LLMs to Follow Zero-shot Demonstrative Instructions8 Aug
  4. 2023MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilities4 Aug
  5. 2023SEED-Bench: Benchmarking Multimodal LLMs with Generative Comprehension30 Jul
  6. 2023MMBench: Is Your Multi-modal Model an All-around Player?12 Jul
  7. 2023A Survey on Multimodal Large Language Models23 Jun
  8. 2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Models23 Jun
  9. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning11 May

Other references · 98

  1. 2019nuScenes: A multimodal dataset for autonomous drivingcited by 4,485
  2. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
  3. 2017ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenescited by 3,088
  4. 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,048
  5. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,689
  6. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,646
  7. 2015VQA: Visual Question Answeringcited by 2,489
  8. 2023Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyondcited by 2,179
  9. 2024Qwen2 Technical Reportcited by 2,076
  10. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,044
  11. 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answeringcited by 1,881
  12. 2021Ego4D: Around the World in 3,000 Hours of Egocentric Videocited by 1,530
38 more
  1. 2022ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoningcited by 1,449
  2. 2016CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoningcited by 1,445
  3. 2023Video-LLaVA: Learning United Visual Representation by Alignment Before Projectioncited by 1,385
  4. 2024Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysiscited by 1,375
  5. 2020DocVQA: A Dataset for VQA on Document Imagescited by 1,288
  6. 2019OK-VQA: A Visual Question Answering Benchmark Requiring External Knowledgecited by 1,114
  7. 2016Modeling Context in Referring Expressionscited by 1,054
  8. 2023Video-ChatGPT: Towards Detailed Video Understanding via Large Vision and Language Modelscited by 1,047
  9. 2024Are We on the Right Way for Evaluating Large Vision-Language Models?cited by 893
  10. 2018VizWiz Grand Challenge: Answering Visual Questions from Blind Peoplecited by 852
  11. 2016A Diagram Is Worth A Dozen Imagescited by 793
  12. 2023LLaMA-Adapter: Efficient Fine-tuning of Language Models with Zero-init Attentioncited by 776
  13. 2019ALFRED: A Benchmark for Interpreting Grounded Instructions for Everyday Taskscited by 773
  14. 2021NExT-QA:Next Phase of Question-Answering to Explaining Temporal Actionscited by 735
  15. 2023The Flan Collection: Designing Data and Methods for Effective Instruction Tuningcited by 734
  16. 2024DeepSeek-VL: Towards Real-World Vision-Language Understandingcited by 718
  17. 2022A-OKVQA: A Benchmark for Visual Question Answering using World Knowledgecited by 702
  18. 2024PaliGemma: A versatile 3B VLM for transfercited by 679
  19. 2023EgoSchema: A Diagnostic Benchmark for Very Long-form Video Language Understandingcited by 666
  20. 2019MathQA: Towards Interpretable Math Word Problem Solving with Operation-Based Formalismscited by 653
  21. 2017Towards Automatic Learning of Procedures from Web Instructional Videoscited by 616
  22. 2023MagicBrush: A Manually Annotated Dataset for Instruction-Guided Image Editingcited by 615
  23. 2024MathVerse: Does Your Multi-modal LLM Truly See the Diagrams in Visual Math Problems?cited by 573
  24. 2019ActivityNet-QA: A Dataset for Understanding Complex Web Videos via Question Answeringcited by 559
  25. 2023LLaMA-VID: An Image is Worth 2 Tokens in Large Language Modelscited by 539
  26. 2016Hollywood in Homes: Crowdsourcing Data Collection for Activity Understandingcited by 535
  27. 2024Long Context Transfer from Language to Visioncited by 502
  28. 2020The Hateful Memes Challenge: Detecting Hate Speech in Multimodal Memescited by 485
  29. 2018A Corpus for Reasoning About Natural Language Grounded in Photographscited by 457
  30. 2021InfographicVQAcited by 430
  31. 2021OCR-free Document Understanding Transformercited by 400
  32. 2020TextCaps: a Dataset for Image Captioning with Reading Comprehensioncited by 398
  33. 2024ShareGPT4Video: Improving Video Understanding and Generation with Better Captionscited by 386
  34. 2021Inter-GPS: Interpretable Geometry Problem Solving with Formal Language and Symbolic Reasoningcited by 376
  35. 2024Mini-Gemini: Mining the Potential of Multi-modality Vision Language Modelscited by 361
  36. 2022Dynamic Prompt Learning via Policy Gradient for Semi-structured Mathematical Reasoningcited by 347
  37. 2024Magpie: Alignment Data Synthesis from Scratch by Prompting Aligned LLMs with Nothingcited by 341
  38. 2021ScanQA: 3D Question Answering for Spatial Scene Understandingcited by 340

50 most cited of 98 All 98 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 238; 2025: 1362; 2026: 1069.

2,669 papers.

Sort:

By area:

  1. 2025InternVL3: Exploring Advanced Training and Test-Time Recipes for Open-Source Multimodal Modelscited by 1,769
  2. 2024Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scalingcited by 1,573
  3. 2025InternVL3.5: Advancing Open-Source Multimodal Models in Versatility, Reasoning, and Efficiencycited by 1,419
  4. 2025Emerging Properties in Unified Multimodal Pretrainingcited by 855
  5. 2024Emu3: Next-Token Prediction is All You Needcited by 761
  6. 2025V-JEPA 2: Self-Supervised Video Models Enable Understanding, Prediction and Planningcited by 738
  7. 2024Thinking in Space: How Multimodal Large Language Models See, Remember, and Recall Spacescited by 580
  8. 2024DeepSeek-VL2: Mixture-of-Experts Vision-Language Models for Advanced Multimodal Understandingcited by 559
  9. 2024LLaVA-Video: Video Instruction Tuning With Synthetic Datacited by 550
  10. 2025VLM-R1: A Stable and Generalizable R1-style Large Vision-Language Modelcited by 515
  11. 2025VideoLLaMA 3: Frontier Multimodal Foundation Models for Image and Video Understandingcited by 502
  12. 2025Visual-RFT: Visual Reinforcement Fine-Tuningcited by 501
38 more
  1. 2024MMMU-Pro: A More Robust Multi-discipline Multimodal Understanding Benchmarkcited by 494
  2. 2025Phi-4-Mini Technical Report: Compact yet Powerful Multimodal Language Models via Mixture-of-LoRAscited by 491
  3. 2025Video-R1: Reinforcing Video Reasoning in MLLMscited by 464
  4. 2024Janus: Decoupling Visual Encoding for Unified Multimodal Understanding and Generationcited by 447
  5. 2025DeepEyes: Incentivizing "Thinking with Images" via Reinforcement Learningcited by 398
  6. 2025Kimi-VL Technical Reportcited by 381
  7. 2025R1-Onevision: Advancing Generalized Multimodal Reasoning through Cross-Modal Formalizationcited by 364
  8. 2025WorldVLA: Towards Autoregressive Action World Modelcited by 351
  9. 2024LongVU: Spatiotemporal Adaptive Compression for Long Video-Language Understandingcited by 341
  10. 2024Molmo and PixMo: Open Weights and Open Data for State-of-the-Art Vision-Language Modelscited by 321
  11. 2025VL-Rethinker: Incentivizing Self-Reflection of Vision-Language Models with Reinforcement Learningcited by 315
  12. 2025Perception Encoder: The best visual embeddings are not at the output of the networkcited by 315
  13. 2024LongVILA: Scaling Long-Context Visual Language Models for Long Videoscited by 303
  14. 2025X-VLA: Soft-Prompted Transformer as Scalable Cross-Embodiment Vision-Language-Action Modelcited by 285
  15. 2024Aguvis: Unified Pure Vision Agents for Autonomous GUI Interactioncited by 271
  16. 2025Pixel Reasoner: Incentivizing Pixel-Space Reasoning with Curiosity-Driven Reinforcement Learningcited by 267
  17. 2025ScreenSpot-Pro: GUI Grounding for Professional High-Resolution Computer Usecited by 246
  18. 2025Show-o2: Improved Native Unified Multimodal Modelscited by 242
  19. 2025OmniGen2: Towards Instruction-Aligned Multimodal Generationcited by 241
  20. 2024Enhancing the Reasoning Ability of Multimodal Large Language Models via Mixed Preference Optimizationcited by 238
  21. 2024VisRAG: Vision-based Retrieval-augmented Generation on Multi-modality Documentscited by 237
  22. 2025LLaVA-OneVision-1.5: Fully Open Framework for Democratized Multimodal Trainingcited by 233
  23. 2025Transfer between Modalities with MetaQueriescited by 231
  24. 2024VideoChat-Flash: Hierarchical Compression for Long-Context Video Modelingcited by 212
  25. 2025SFT or RL? An Early Investigation into Training R1-Like Reasoning Large Vision-Language Modelscited by 209
  26. 2024NVILA: Efficient Frontier Visual Language Modelscited by 206
  27. 2025Spatial-MLLM: Boosting MLLM Capabilities in Visual-based Spatial Intelligencecited by 206
  28. 2024MetaMorph: Multimodal Understanding and Generation via Instruction Tuningcited by 205
  29. 2025InternVideo2.5: Empowering Video MLLMs with Long and Rich Context Modelingcited by 203
  30. 2025Unified Reward Model for Multimodal Understanding and Generationcited by 194
  31. 2025VITA-1.5: Towards GPT-4o Level Real-Time Vision and Speech Interactioncited by 191
  32. 2024LLaVA-3D: A Simple yet Effective Pathway to Empowering LMMs with 3D-awarenesscited by 178
  33. 2025Multimodal Chain-of-Thought Reasoning: A Comprehensive Surveycited by 176
  34. 2024Pixtral 12Bcited by 174
  35. 2025MLLMs Know Where to Look: Training-free Perception of Small Visual Details with Multimodal LLMscited by 172
  36. 2024Building and better understanding vision-language models: insights and future directionscited by 169
  37. 2025MMSI-Bench: A Benchmark for Multi-Image Spatial Intelligencecited by 169
  38. 2024OCRBench v2: An Improved Benchmark for Evaluating Large Multimodal Models on Visual Text Localization and Reasoningcited by 165

50 most cited of 2,669 All 2,669 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025LLaVA-Mini: Efficient Image and Video Large Multimodal Models with One Vision Token
  2. 2024LLaVA-NeXT-Interleave: Tackling Multi-image, Video, and 3D in Large Multimodal Models
  3. 2024LLaVA-UHD: an LMM Perceiving Any Aspect Ratio and High-Resolution Images
  4. 2025Rethinking Visual Information Processing in Multimodal LLMs
  5. 2024AVG-LLaVA: An Efficient Large Multimodal Model with Adaptive Visual Granularity
  6. 2023LLaVA-Plus: Learning to Use Tools for Creating Multimodal Agents
  7. 2024TinyLLaVA: A Framework of Small-scale Large Multimodal Models
  8. 2024TC-LLaVA: Rethinking the Transfer from Image to Video Understanding with Temporal Considerations