mlsanityData to 7 Oct 2026

LLaVA-NeXT-Interleave: Tackling Multi-image, Video, and 3D in Large Multimodal Models

arXiv 2407.07895 ↗First posted 10 Jul 2024cs.CV cs.CL cs.LGcited by 571

On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning

Abstract (arXiv)

Visual instruction tuning has made considerable strides in enhancing the capabilities of Large Multimodal Models (LMMs). However, existing open LMMs largely focus on single-image tasks, their applications to multi-image scenarios remains less explored. Additionally, prior LMM research separately tackles different scenarios, leaving it impossible to generalize cross scenarios with new emerging capabilities. To this end, we introduce LLaVA-NeXT-Interleave, which simultaneously tackles Multi-image, Multi-frame (video), Multi-view (3D), and Multi-patch (single-image) scenarios in LMMs. To enable these capabilities, we regard the interleaved data format as a general template and compile the M4-Instruct dataset with 1,177.6k samples, spanning 4 primary domains with 14 tasks and 41 datasets. We also curate the LLaVA-Interleave Bench to comprehensively evaluate the multi-image performance of LMMs. Through extensive experiments, LLaVA-NeXT-Interleave achieves leading results in multi-image, video, and 3D benchmarks, while maintaining the performance of single-image tasks. Besides, our model also exhibits several emerging capabilities, e.g., transferring tasks across different settings and modalities. Code is available at https://github.com/LLaVA-VL/LLaVA-NeXT

Key facts. Cited by 571 papers in the map (data through 7 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 49 of its 57 references that are in the map. First posted on arXiv 10 Jul 2024.

Builds on

49 of 57 references.

Closest ancestors · 10

Same direction, newest first

  1. 2024MuirBench: A Comprehensive Benchmark for Robust Multi-image Understanding13 Jun
  2. 2024BLINK: Multimodal Large Language Models Can See but Not Perceive18 Apr
  3. 2024Direct Preference Optimization of Video Large Multimodal Models from Language Model Reward1 Apr
  4. 2024SPHINX-X: Scaling Data and Parameters for a Family of Multi-modal Large Language Models8 Feb
  5. 2023Generative Multimodal Models are In-Context Learners20 Dec
  6. 2023VILA: On Pre-training for Visual Language Models12 Dec
  7. 2023SPHINX: The Joint Mixing of Weights, Tasks, and Visual Embeddings for Multi-modal Large Language Models13 Nov
  8. 2023Beyond Task Performance: Evaluating and Reducing the Flaws of Large Multimodal Models with In-Context Learning1 Oct
  9. 2023Fine-tuning Multimodal LLMs to Follow Zero-shot Demonstrative Instructions8 Aug
  10. 2023OBELICS: An Open Web-Scale Filtered Dataset of Interleaved Image-Text Documents21 Jun

Other references · 39

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  4. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  5. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  6. 2023Visual Instruction Tuningcited by 7,792
  7. 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,764
  8. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  9. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,310
  10. 2023Qwen Technical Reportcited by 3,273
  11. 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,048
  12. 2023Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyondcited by 2,179
27 more
  1. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,044
  2. 2021LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairscited by 1,544
  3. 2023MVBench: A Comprehensive Multi-modal Video Understanding Benchmarkcited by 1,086
  4. 2023Video-ChatGPT: Towards Detailed Video Understanding via Large Vision and Language Modelscited by 1,047
  5. 2023VideoChat: Chat-Centric Video Understandingcited by 851
  6. 2019ALFRED: A Benchmark for Interpreting Grounded Instructions for Everyday Taskscited by 773
  7. 2021NExT-QA:Next Phase of Question-Answering to Explaining Temporal Actionscited by 735
  8. 2024MathVerse: Does Your Multi-modal LLM Truly See the Diagrams in Visual Math Problems?cited by 573
  9. 2023OpenFlamingo: An Open-Source Framework for Training Large Autoregressive Vision-Language Modelscited by 560
  10. 2019ActivityNet-QA: A Dataset for Understanding Complex Web Videos via Question Answeringcited by 559
  11. 2023Language Is Not All You Need: Aligning Perception with Language Modelscited by 543
  12. 2023LLaMA-VID: An Image is Worth 2 Tokens in Large Language Modelscited by 539
  13. 2018A Corpus for Reasoning About Natural Language Grounded in Photographscited by 457
  14. 2021ScanQA: 3D Question Answering for Spatial Scene Understandingcited by 340
  15. 2023PointLLM: Empowering Large Language Models to Understand Point Cloudscited by 326
  16. 2024What matters when building vision-language models?cited by 321
  17. 2023MIMIC-IT: Multi-Modal In-Context Instruction Tuningcited by 264
  18. 2023Q-Bench: A Benchmark for General-Purpose Foundation Models on Low-level Visioncited by 247
  19. 2024STAR: A Benchmark for Situated Reasoning in Real-World Videoscited by 213
  20. 2023Multimodal C4: An Open, Billion-scale Corpus of Images Interleaved with Textcited by 208
  21. 2023Point-Bind & Point-LLM: Aligning Point Cloud with Multi-modality for 3D Understanding, Generation, and Instruction Followingcited by 192
  22. 2023ImageBind-LLM: Multi-modality Instruction Tuningcited by 142
  23. 2023VideoLLM: Modeling Video Sequence with Large Language Modelscited by 116
  24. 2023Vista-LLaMA: Reducing Hallucination in Video Language Models via Equal Distance to Visual Tokenscited by 70
  25. 2024Many-Shot In-Context Learning in Multimodal Foundation Modelscited by 64
  26. 2024ReMI: A Dataset for Reasoning with Multiple Imagescited by 27
  27. 2020Visual Question Answering on Image Setscited by 26

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 87; 2025: 307; 2026: 177.

571 papers.

Sort:

By area:

  1. 2024DeepSeek-VL2: Mixture-of-Experts Vision-Language Models for Advanced Multimodal Understandingcited by 559
  2. 2025VideoLLaMA 3: Frontier Multimodal Foundation Models for Image and Video Understandingcited by 502
  3. 2024MMMU-Pro: A More Robust Multi-discipline Multimodal Understanding Benchmarkcited by 494
  4. 2025VBench-2.0: Advancing Video Generation Benchmark Suite for Intrinsic Faithfulnesscited by 289
  5. 2024mPLUG-Owl3: Towards Long Image-Sequence Understanding in Multi-Modal Large Language Modelscited by 285
  6. 2024VideoChat-Flash: Hierarchical Compression for Long-Context Video Modelingcited by 212
  7. 2025InternVideo2.5: Empowering Video MLLMs with Long and Rich Context Modelingcited by 203
  8. 2025HybridVLA: Collaborative Diffusion and Autoregression in a Unified Vision-Language-Action Modelcited by 176
  9. 2025Multimodal Chain-of-Thought Reasoning: A Comprehensive Surveycited by 176
  10. 2024Towards World Simulator: Crafting Physical Commonsense-Based Benchmark for Video Generationcited by 167
  11. 2024Oryx MLLM: On-Demand Spatial-Temporal Understanding at Arbitrary Resolutioncited by 157
  12. 2024VisionReward: Fine-Grained Multi-Dimensional Human Preference Learning for Image and Video Generationcited by 148
38 more
  1. 2025PhysBench: Benchmarking and Enhancing Vision-Language Models for Physical World Understandingcited by 147
  2. 2024StreamingBench: Assessing the Gap for MLLMs to Achieve Streaming Video Understandingcited by 135
  3. 2025RoboBrain: A Unified Brain Model for Robotic Manipulation from Abstract to Concretecited by 134
  4. 2024Kangaroo: A Powerful Video-Language Model Supporting Long-context Video Inputcited by 132
  5. 2025MMVU: Measuring Expert-Level Multi-Discipline Video Understandingcited by 131
  6. 2025MME-CoT: Benchmarking Chain-of-Thought in Large Multimodal Models for Reasoning Quality, Robustness, and Efficiencycited by 123
  7. 2024mPLUG-DocOwl2: High-resolution Compressing for OCR-free Multi-page Document Understandingcited by 113
  8. 2024MAVIS: Mathematical Visual Instruction Tuning with an Automatic Data Enginecited by 111
  9. 2025GR-3 Technical Reportcited by 110
  10. 2024LamRA: Large Multimodal Model as Your Advanced Retrieval Assistantcited by 106
  11. 2025Daily-Omni: Towards Audio-Visual Reasoning with Temporal Alignment across Modalitiescited by 101
  12. 2024MMSearch: Benchmarking the Potential of Large Models as Multi-modal Search Enginescited by 98
  13. 2024LayoutVLM: Differentiable Optimization of 3D Layout via Vision-Language Modelscited by 94
  14. 2024DyCoke: Dynamic Compression of Tokens for Fast Video Large Language Modelscited by 93
  15. 2024OpenEMMA: Open-Source Multimodal Model for End-to-End Autonomous Drivingcited by 93
  16. 2025ChartCoder: Advancing Multimodal Large Language Model for Chart-to-Code Generationcited by 88
  17. 2025SpatialLadder: Progressive Training for Spatial Reasoning in Vision-Language Modelscited by 83
  18. 2024Frame-Voyager: Learning to Query Frames for Video Large Language Modelscited by 80
  19. 2025Zebra-CoT: A Dataset for Interleaved Vision Language Reasoningcited by 78
  20. 2024MM1.5: Methods, Analysis & Insights from Multimodal LLM Fine-tuningcited by 75
  21. 2025Kwai Keye-VL 1.5 Technical Reportcited by 68
  22. 2024Infinity-MM: Scaling Multimodal Performance with Large-Scale and High-Quality Instruction Datacited by 67
  23. 2024VideoEspresso: A Large-Scale Chain-of-Thought Dataset for Fine-Grained Video Reasoning via Core Frame Selectioncited by 64
  24. 2025Spatial Reasoning with Vision-Language Models in Ego-Centric Multi-View Scenescited by 62
  25. 2024MMIU: Multimodal Multi-image Understanding for Evaluating Large Vision-Language Modelscited by 61
  26. 2025HumanOmni: A Large Vision-Speech Language Model for Human-Centric Video Understandingcited by 58
  27. 2025DiTAR: Diffusion Transformer Autoregressive Modeling for Speech Generationcited by 56
  28. 2025LION-FS: Fast & Slow Video-Language Thinker as Online Video Assistantcited by 56
  29. 2025UniCTokens: Boosting Personalized Understanding and Generation via Unified Concept Tokenscited by 55
  30. 2025Towards Zero-Shot Anomaly Detection and Reasoning with Multimodal Large Language Modelscited by 54
  31. 2025Visual Agentic Reinforcement Fine-Tuningcited by 53
  32. 2025SmolDocling: An ultra-compact vision-language model for end-to-end multi-modal document conversioncited by 51
  33. 2025Improved Visual-Spatial Reasoning via R1-Zero-Like Trainingcited by 51
  34. 2025Advancing Multimodal Reasoning: From Optimized Cold Start to Staged Reinforcement Learningcited by 51
  35. 2025VLAS: Vision-Language-Action Model With Speech Instructions For Customized Robot Manipulationcited by 48
  36. 2025OmniMMI: A Comprehensive Multi-modal Interaction Benchmark in Streaming Video Contextscited by 48
  37. 2025DyFo: A Training-Free Dynamic Focus Visual Search for Enhancing LMMs in Fine-Grained Visual Understandingcited by 47
  38. 2025Kwai Keye-VL Technical Reportcited by 47

50 most cited of 571 All 571 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2023An Empirical Study of Scaling Instruct-Tuned Large Multimodal Models
  2. 2023Improved Baselines with Visual Instruction Tuning
  3. 2024LLaVA-OneVision: Easy Visual Task Transfer
  4. 2025LLaVA-MORE: A Comparative Study of LLMs and Visual Backbones for Enhanced Visual Instruction Tuning
  5. 2025LLaVA-Mini: Efficient Image and Video Large Multimodal Models with One Vision Token
  6. 2025LLaVA-OneVision-1.5: Fully Open Framework for Democratized Multimodal Training
  7. 2024LLaVA-Video: Video Instruction Tuning With Synthetic Data
  8. 2024LLaVA-UHD: an LMM Perceiving Any Aspect Ratio and High-Resolution Images