mlsanityData to 7 Oct 2026

mPLUG-Owl3: Towards Long Image-Sequence Understanding in Multi-Modal Large Language Models

arXiv 2408.04840 ↗First posted 9 Aug 2024cs.CV cs.AI cs.CLcited by 285

On the map: Multimodal LLMs / vision-language models → Video understanding with MLLMs

Abstract (arXiv)

Multi-modal Large Language Models (MLLMs) have demonstrated remarkable capabilities in executing instructions for a variety of single-image tasks. Despite this progress, significant challenges remain in modeling long image sequences. In this work, we introduce the versatile multi-modal large language model, mPLUG-Owl3, which enhances the capability for long image-sequence understanding in scenarios that incorporate retrieved image-text knowledge, interleaved image-text, and lengthy videos. Specifically, we propose novel hyper attention blocks to efficiently integrate vision and language into a common language-guided semantic space, thereby facilitating the processing of extended multi-image scenarios. Extensive experimental results suggest that mPLUG-Owl3 achieves state-of-the-art performance among models with a similar size on single-image, multi-image, and video benchmarks. Moreover, we propose a challenging long visual sequence evaluation named Distractor Resistance to assess the ability of models to maintain focus amidst distractions. Finally, with the proposed architecture, mPLUG-Owl3 demonstrates outstanding performance on ultra-long visual sequence inputs. We hope that mPLUG-Owl3 can contribute to the development of more efficient and powerful multimodal large language models.

Key facts. Cited by 285 papers in the map (data through 7 Oct 2026). Placed in Video understanding with MLLMs, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 66 of its 76 references that are in the map. First posted on arXiv 9 Aug 2024.

Builds on

66 of 76 references.

Closest ancestors · 6

Same direction, newest first

  1. 2024LongVideoBench: A Benchmark for Long-context Interleaved Video-Language Understanding22 Jul
  2. 2024InternLM-XComposer-2.5: A Versatile Large Vision Language Model Supporting Long-Contextual Input and Output3 Jul
  3. 2024Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysis31 May
  4. 2024PLLaVA : Parameter-free LLaVA Extension from Images to Videos for Video Dense Captioning25 Apr
  5. 2023MVBench: A Comprehensive Multi-modal Video Understanding Benchmark28 Nov
  6. 2023Video-LLaVA: Learning United Visual Representation by Alignment Before Projection16 Nov

Other references · 60

  1. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  2. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  4. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  5. 2023Visual Instruction Tuningcited by 7,792
  6. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  7. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
  8. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  9. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,131
  10. 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,048
  11. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,855
  12. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,689
38 more
  1. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Taskscited by 2,534
  2. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,498
  3. 2023MMBench: Is Your Multi-modal Model an All-around Player?cited by 2,105
  4. 2024Qwen2 Technical Reportcited by 2,076
  5. 2019Towards VQA Models That Can Readcited by 1,696
  6. 2023Evaluating Object Hallucination in Large Vision-Language Modelscited by 1,696
  7. 2023MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilitiescited by 1,221
  8. 2021Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Conceptscited by 1,209
  9. 2019OK-VQA: A Visual Question Answering Benchmark Requiring External Knowledgecited by 1,114
  10. 2024MiniCPM-V: A GPT-4V Level MLLM on Your Phonecited by 1,061
  11. 2023mPLUG-Owl: Modularization Empowers Large Language Models with Multimodalitycited by 1,038
  12. 2023VideoChat: Chat-Centric Video Understandingcited by 851
  13. 2024Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMscited by 822
  14. 2016A Diagram Is Worth A Dozen Imagescited by 793
  15. 2024VideoLLaMA 2: Advancing Spatial-Temporal Modeling and Audio Understanding in Video-LLMscited by 749
  16. 2021NExT-QA:Next Phase of Question-Answering to Explaining Temporal Actionscited by 735
  17. 2024DeepSeek-VL: Towards Real-World Vision-Language Understandingcited by 718
  18. 2023CogVLM: Visual Expert for Pretrained Language Modelscited by 704
  19. 2023VILA: On Pre-training for Visual Language Modelscited by 686
  20. 2023DataComp: In search of the next generation of multimodal datasetscited by 635
  21. 2024LLaVA-NeXT-Interleave: Tackling Multi-image, Video, and 3D in Large Multimodal Modelscited by 571
  22. 2023OpenFlamingo: An Open-Source Framework for Training Large Autoregressive Vision-Language Modelscited by 560
  23. 2023mPLUG-Owl2: Revolutionizing Multi-modal Large Language Model with Modality Collaborationcited by 545
  24. 2024BLINK: Multimodal Large Language Models Can See but Not Perceivecited by 529
  25. 2018A Corpus for Reasoning About Natural Language Grounded in Photographscited by 457
  26. 2019VATEX: A Large-Scale, High-Quality Multilingual Dataset for Video-and-Language Researchcited by 412
  27. 2024ShareGPT4Video: Improving Video Understanding and Generation with Better Captionscited by 386
  28. 2024Mini-Gemini: Mining the Potential of Multi-modality Vision Language Modelscited by 361
  29. 2024What matters when building vision-language models?cited by 321
  30. 2023InternLM-XComposer: A Vision-Language Large Model for Advanced Text-image Comprehension and Compositioncited by 297
  31. 2021IconQA: A New Benchmark for Abstract Diagram Understanding and Visual Language Reasoningcited by 262
  32. 2022MuRAG: Multimodal Retrieval-Augmented Generator for Open Question Answering over Images and Textcited by 216
  33. 2024MANTIS: Interleaved Multi-Image Instruction Tuningcited by 212
  34. 2024ALLaVA: Harnessing GPT4V-Synthesized Data for Lite Vision-Language Modelscited by 204
  35. 2024Mobile-Agent-v2: Mobile Device Operation Assistant with Effective Navigation via Multi-Agent Collaborationcited by 187
  36. 2023MMICL: Empowering Vision-language Model with Multi-Modal In-Context Learningcited by 171
  37. 2024UFO: A UI-Focused Agent for Windows OS Interactioncited by 147
  38. 2024Direct Preference Optimization of Video Large Multimodal Models from Language Model Rewardcited by 145

50 most cited of 60 All 60 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 44; 2025: 169; 2026: 72.

285 papers.

Sort:

By area:

  1. 2025VideoLLaMA 3: Frontier Multimodal Foundation Models for Image and Video Understandingcited by 502
  2. 2025DeepEyes: Incentivizing "Thinking with Images" via Reinforcement Learningcited by 398
  3. 2024PyramidDrop: Accelerating Your Large Vision-Language Models via Pyramid Visual Redundancy Reductioncited by 283
  4. 2025LMM-R1: Empowering 3B LMMs with Strong Reasoning Abilities Through Two-Stage Rule-Based RLcited by 244
  5. 2025InternVideo2.5: Empowering Video MLLMs with Long and Rich Context Modelingcited by 203
  6. 2024OCRBench v2: An Improved Benchmark for Evaluating Large Multimodal Models on Visual Text Localization and Reasoningcited by 165
  7. 2025LLaVA-Mini: Efficient Image and Video Large Multimodal Models with One Vision Tokencited by 164
  8. 2025Sa2VA: Marrying SAM2 with MLLM for Dense Grounded Understanding of Images and Videoscited by 154
  9. 2025PhysBench: Benchmarking and Enhancing Vision-Language Models for Physical World Understandingcited by 147
  10. 2025WorldSense: Evaluating Real-world Omnimodal Understanding for Multimodal LLMscited by 147
  11. 2024mPLUG-DocOwl2: High-resolution Compressing for OCR-free Multi-page Document Understandingcited by 113
  12. 2025VisuLogic: A Benchmark for Evaluating Visual Reasoning in Multi-modal Large Language Modelscited by 102
38 more
  1. 2024MMSearch: Benchmarking the Potential of Large Models as Multi-modal Search Enginescited by 98
  2. 2024Mini-InternVL: A Flexible-Transfer Pocket Multimodal Model with 5% Parameters and 90% Performancecited by 94
  3. 2025Deep Video Discovery: Agentic Search with Tool Use for Long-form Video Understandingcited by 94
  4. 2025EgoLife: Towards Egocentric Life Assistantcited by 86
  5. 2025VideoRFT: Incentivizing Video Reasoning Capability in MLLMs via Reinforced Fine-Tuningcited by 86
  6. 2025UniVG-R1: Reasoning Guided Universal Visual Grounding with Reinforcement Learningcited by 83
  7. 2024FastVLM: Efficient Vision Encoding for Vision Language Modelscited by 76
  8. 2025BOLT: Boost Large Vision-Language Model Without Training for Long-form Video Understandingcited by 72
  9. 2024VideoEspresso: A Large-Scale Chain-of-Thought Dataset for Fine-Grained Video Reasoning via Core Frame Selectioncited by 64
  10. 2025LION-FS: Fast & Slow Video-Language Thinker as Online Video Assistantcited by 56
  11. 2024InternLM-XComposer2.5-OmniLive: A Comprehensive Multimodal System for Long-term Streaming Video and Audio Interactionscited by 48
  12. 2024Nullu: Mitigating Object Hallucinations in Large Vision-Language Models via HalluSpace Projectioncited by 46
  13. 2025STORM: Token-Efficient Long Video Understanding for Multimodal LLMscited by 46
  14. 2024AIM: Adaptive Inference of Multi-Modal LLMs via Token Merging and Pruningcited by 43
  15. 2025Open-o3-Video: Grounded Video Reasoning with Explicit Spatio-Temporal Evidencecited by 43
  16. 2024MRAG-Bench: Vision-Centric Evaluation for Retrieval-Augmented Multimodal Modelscited by 41
  17. 2024Q-Bench-Video: Benchmarking the Video Quality Understanding of LMMscited by 39
  18. 2025Long-VITA: Scaling Large Multi-modal Models to 1 Million Tokens with Leading Short-Context Accuracycited by 39
  19. 2024Core Knowledge Deficits in Multi-Modal Language Modelscited by 38
  20. 2024EvalMuse-40K: A Reliable and Fine-Grained Benchmark with Comprehensive Human Annotations for Text-to-Image Generation Model Evaluationcited by 37
  21. 2025Video-MMLU: A Massive Multi-Discipline Lecture Understanding Benchmarkcited by 37
  22. 2024Lost in Time: A New Temporal Benchmark for VideoLLMscited by 36
  23. 2025POSTA: A Go-to Framework for Customized Artistic Poster Generationcited by 36
  24. 2025Visual Large Language Models for Generalized and Specialized Applicationscited by 35
  25. 2025Vamba: Understanding Hour-Long Videos with Hybrid Mamba-Transformerscited by 35
  26. 2024ReTaKe: Reducing Temporal and Knowledge Redundancy for Long Video Understandingcited by 33
  27. 2025Scaling Video-Language Models to 10K Frames via Hierarchical Differential Distillationcited by 33
  28. 2025Evaluating and Steering Modality Preferences in Multimodal Large Language Modelcited by 33
  29. 2024VidEgoThink: Assessing Egocentric Video Understanding Capabilities for Embodied AIcited by 30
  30. 2025SurgVLM: A Large Vision-Language Model and Systematic Evaluation Benchmark for Surgical Intelligencecited by 30
  31. 2024R-Bench: Are your Large Multimodal Model Robust to Real-world Corruptions?cited by 29
  32. 2025VideoITG: Multimodal Video Understanding with Instructed Temporal Groundingcited by 28
  33. 2025Video-SafetyBench: A Benchmark for Safety Evaluation of Video LVLMscited by 25
  34. 2025Reinforcement Learning Tuning for VideoLLMs: Reward Design and Data Efficiencycited by 25
  35. 2025LLaVA-Scissor: Token Compression with Semantic Connected Components for Video LLMscited by 25
  36. 2025VideoARM: Agentic Reasoning over Hierarchical Memory for Long-Form Video Understandingcited by 25
  37. 2024SPORTU: A Comprehensive Sports Understanding Benchmark for Multimodal Large Language Modelscited by 24
  38. 2025LMM4LMM: Benchmarking and Evaluating Large-multimodal Image Generation with LMMscited by 24

50 most cited of 285 All 285 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024From Seconds to Hours: Reviewing MultiModal Large Language Models on Comprehensive Long Video Understanding
  2. 2022mPLUG: Effective and Efficient Vision-Language Learning by Cross-modal Skip-connections
  3. 2024LongVLM: Efficient Long Video Understanding via Large Language Models
  4. 2024Visual Context Window Extension: A New Perspective for Long Video Understanding
  5. 2024B-VLLM: A Vision Large Language Model with Balanced Spatio-Temporal Tokens
  6. 2025Video-XL-2: Towards Very Long-Video Understanding Through Task-Aware KV Sparsification
  7. 2026Attend Before Attention: Efficient and Scalable Video Understanding via Autoregressive Gazing
  8. 2024LongLLaVA: Scaling Multi-modal LLMs to 1000 Images Efficiently via a Hybrid Architecture