mlsanityData to 7 Oct 2026

LongLLaVA: Scaling Multi-modal LLMs to 1000 Images Efficiently via a Hybrid Architecture

arXiv 2409.02889 ↗First posted 4 Sep 2024cs.CL cs.AI cs.CVcited by 100

On the map: Multimodal LLMs / vision-language models → Video understanding with MLLMs

Abstract (arXiv)

Expanding the long-context capabilities of Multi-modal Large Language Models~(MLLMs) is critical for advancing video understanding and high-resolution image analysis. Achieving this requires systematic improvements in model architecture, data construction, and training strategies, particularly to address challenges such as performance degradation with increasing image counts and high computational costs. In this paper, we propose a hybrid architecture that integrates Mamba and Transformer blocks, introduce data construction methods that capture both temporal and spatial dependencies, and employ a progressive training strategy. Our released model, LongLLaVA (Long-Context Large Language and Vision Assistant), demonstrates an effective balance between efficiency and performance. LongLLaVA achieves competitive results across various benchmarks while maintaining high throughput and low memory consumption. Notably, it can process nearly one thousand images on a single A100 80GB GPU, underscoring its potential for a wide range of multi-modal applications.

Key facts. Cited by 100 papers in the map (data through 7 Oct 2026). Placed in Video understanding with MLLMs, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 55 of its 90 references that are in the map. First posted on arXiv 4 Sep 2024.

Builds on

55 of 90 references.

Closest ancestors · 8

Same direction, newest first

  1. 2024LongVILA: Scaling Long-Context Visual Language Models for Long Videos19 Aug
  2. 2024LongVideoBench: A Benchmark for Long-context Interleaved Video-Language Understanding22 Jul
  3. 2024OmChat: A Recipe to Train Multimodal Language Models with Strong Long Context and Video Understanding6 Jul
  4. 2024Long Context Transfer from Language to Vision24 Jun
  5. 2024Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysis31 May
  6. 2024PLLaVA : Parameter-free LLaVA Extension from Images to Videos for Video Dense Captioning25 Apr
  7. 2023MVBench: A Comprehensive Multi-modal Video Understanding Benchmark28 Nov
  8. 2023Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understanding5 Jun

Other references · 47

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  3. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  4. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
  5. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
  6. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,018
  7. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,175
  8. 2024LLaVA-OneVision: Easy Visual Task Transfercited by 2,669
  9. 2019BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questionscited by 2,140
  10. 2023MMBench: Is Your Multi-modal Model an All-around Player?cited by 2,105
  11. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,044
  12. 2024Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phonecited by 1,883
35 more
  1. 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answeringcited by 1,881
  2. 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,838
  3. 2022Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Themcited by 1,767
  4. 2020GLU Variants Improve Transformercited by 1,724
  5. 2022ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoningcited by 1,449
  6. 2019Root Mean Square Layer Normalizationcited by 1,425
  7. 2023GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpointscited by 1,311
  8. 2020DocVQA: A Dataset for VQA on Document Imagescited by 1,288
  9. 2024Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Dualitycited by 1,283
  10. 2023MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilitiescited by 1,221
  11. 2024The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scalecited by 1,104
  12. 2024How Far Are We to GPT-4V? Closing the Gap to Commercial Multimodal Models with Open-Source Suitescited by 1,033
  13. 2023ShareGPT4V: Improving Large Multi-Modal Models with Better Captionscited by 1,000
  14. 2022In-context Learning and Induction Headscited by 930
  15. 2023SEED-Bench: Benchmarking Multimodal LLMs with Generative Comprehensioncited by 929
  16. 2024VideoLLaMA 2: Advancing Spatial-Temporal Modeling and Audio Understanding in Video-LLMscited by 749
  17. 2023NExT-GPT: Any-to-Any Multimodal LLMcited by 610
  18. 2024WildChat: 1M ChatGPT Interaction Logs in the Wildcited by 598
  19. 2020PathVQA: 30000+ Questions for Medical Visual Question Answeringcited by 467
  20. 2023V*: Guided Visual Search as a Core Mechanism in Multimodal LLMscited by 459
  21. 2024Jamba: A Hybrid Transformer-Mamba Language Modelcited by 390
  22. 2024ShareGPT4Video: Improving Video Understanding and Generation with Better Captionscited by 386
  23. 2023What Makes Good Data for Alignment? A Comprehensive Study of Automatic Data Selection in Instruction Tuningcited by 354
  24. 2024InternLM-XComposer2: Mastering Free-form Text-Image Composition and Comprehension in Vision-Language Large Modelcited by 326
  25. 2023Multimodal Foundation Models: From Specialists to General-Purpose Assistantscited by 247
  26. 2024MANTIS: Interleaved Multi-Image Instruction Tuningcited by 212
  27. 2024World Model on Million-Length Video And Language With Blockwise RingAttentioncited by 196
  28. 2024HuatuoGPT-Vision, Towards Injecting Medical Visual Knowledge into Multimodal LLMs at Scalecited by 194
  29. 2024Data Engineering for Scaling Language Models to 128K Contextcited by 183
  30. 2024The Mamba in the Llama: Distilling and Accelerating Hybrid Modelscited by 120
  31. 2024Can Mamba Learn How to Learn? A Comparative Study on In-Context Learning Taskscited by 104
  32. 2024SkySenseGPT: A Fine-Grained Instruction Tuning Dataset and Model for Remote Sensing Vision-Language Understandingcited by 101
  33. 2024MileBench: Benchmarking MLLMs in Long Contextcited by 86
  34. 2024Towards Vision-Language Geo-Foundation Model: A Surveycited by 36
  35. 2024STAR: A First-Ever Dataset and A Large-Scale Benchmark for Scene Graph Generation in Large-Size Satellite Imagerycited by 31

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 22; 2025: 61; 2026: 17.

100 papers.

Sort:

By area:

  1. 2025VideoLLaMA 3: Frontier Multimodal Foundation Models for Image and Video Understandingcited by 502
  2. 2025Pixel Reasoner: Incentivizing Pixel-Space Reasoning with Curiosity-Driven Reinforcement Learningcited by 267
  3. 2024VideoChat-Flash: Hierarchical Compression for Long-Context Video Modelingcited by 212
  4. 2024Video-XL: Extra-Long Vision Language Model for Hour-Scale Video Understandingcited by 210
  5. 2025InternVideo2.5: Empowering Video MLLMs with Long and Rich Context Modelingcited by 203
  6. 2025WorldSense: Evaluating Real-world Omnimodal Understanding for Multimodal LLMscited by 147
  7. 2025Magma: A Foundation Model for Multimodal AI Agentscited by 138
  8. 2024Video-RAG: Visually-aligned Retrieval-Augmented Long Video Comprehensioncited by 134
  9. 2024TimeSuite: Improving MLLMs for Long Video Understanding via Grounded Tuningcited by 99
  10. 2025StreamForest: Efficient Online Video Understanding with Persistent Event Memorycited by 71
  11. 2024TimeMarker: A Versatile Video-LLM for Long and Short Video Understanding with Superior Temporal Localization Abilitycited by 69
  12. 2025LVAgent: Long Video Understanding by Multi-Round Dynamical Collaboration of MLLM Agentscited by 58
38 more
  1. 2025SVBench: A Benchmark with Temporal Multi-Turn Dialogues for Streaming Video Understandingcited by 55
  2. 2025ReAgent-V: A Reward-Driven Multi-Agent Framework for Video Understandingcited by 50
  3. 2025WorldMM: Dynamic Multimodal Memory Agent for Long Video Reasoningcited by 50
  4. 2025Video-XL-Pro: Reconstructive Token Compression for Extremely Long Video Understandingcited by 49
  5. 2025OmniMMI: A Comprehensive Multi-modal Interaction Benchmark in Streaming Video Contextscited by 48
  6. 2025STORM: Token-Efficient Long Video Understanding for Multimodal LLMscited by 46
  7. 2024AIM: Adaptive Inference of Multi-Modal LLMs via Token Merging and Pruningcited by 43
  8. 2025Long-VITA: Scaling Large Multi-modal Models to 1 Million Tokens with Leading Short-Context Accuracycited by 39
  9. 2025Video-MMLU: A Massive Multi-Discipline Lecture Understanding Benchmarkcited by 37
  10. 2025Video-XL-2: Towards Very Long-Video Understanding Through Task-Aware KV Sparsificationcited by 36
  11. 2025When Large Vision-Language Model Meets Large Remote Sensing Imagery: Coarse-to-Fine Text-Guided Token Pruningcited by 35
  12. 2025Generative Frame Sampler for Long Video Understandingcited by 35
  13. 2025Vamba: Understanding Hour-Long Videos with Hybrid Mamba-Transformerscited by 35
  14. 2025First SFT, Second RL, Third UPT: Continual Improving Multi-Modal LLM Reasoning via Unsupervised Post-Trainingcited by 34
  15. 2024ReTaKe: Reducing Temporal and Knowledge Redundancy for Long Video Understandingcited by 33
  16. 2025VideoChat-A1: Thinking with Long Videos by Chain-of-Shot Reasoningcited by 32
  17. 2024Dynamic-VLM: Simple Dynamic Visual Token Compression for VideoLLMcited by 31
  18. 2025MEDA: Dynamic KV Cache Allocation for Efficient Multimodal Long-Context Inferencecited by 31
  19. 2025Mavors: Multi-granularity Video Representation for Multimodal Large Language Modelcited by 31
  20. 2024From Seconds to Hours: Reviewing MultiModal Large Language Models on Comprehensive Long Video Understandingcited by 29
  21. 2025MMLongBench: Benchmarking Long-Context Vision-Language Models Effectively and Thoroughlycited by 29
  22. 2024V2PE: Improving Multimodal Long-Context Capability of Vision-Language Models with Variable Visual Position Encodingcited by 28
  23. 2025CoS: Chain-of-Shot Prompting for Long Video Understandingcited by 27
  24. 2025MMInference: Accelerating Pre-filling for Long-Context VLMs via Modality-Aware Permutation Sparse Attentioncited by 26
  25. 2024Inference Optimal VLMs Need Fewer Visual Tokens and More Parameterscited by 25
  26. 2024PVC: Progressive Visual Token Compression for Unified Image and Video Processing in Large Vision-Language Modelscited by 25
  27. 2025MR. Video: "MapReduce" is the Principle for Long Video Understandingcited by 23
  28. 2025LiveStar: Live Streaming Assistant for Real-World Online Video Understandingcited by 22
  29. 2025VideoEval-Pro: Robust and Realistic Long Video Understanding Evaluationcited by 21
  30. 2024VidComposition: Can MLLMs Analyze Compositions in Compiled Videos?cited by 18
  31. 2025Memory-enhanced Retrieval Augmentation for Long Video Understandingcited by 17
  32. 2024VISTA: Enhancing Long-Duration and High-Resolution Video Understanding by Video Spatiotemporal Augmentationcited by 14
  33. 2026From Verbatim to Gist: Distilling Pyramidal Multimodal Memory via Semantic Information Bottleneck for Long-Horizon Video Agentscited by 14
  34. 2024PRIMA: Multi-Image Vision-Language Models for Reasoning Segmentationcited by 13
  35. 2024MAP: Unleashing Hybrid Mamba-Transformer Vision Backbone's Potential with Masked Autoregressive Pretrainingcited by 12
  36. 2025CrossLMM: Decoupling Long Video Sequences from LMMs via Dual Cross-Attention Mechanismscited by 11
  37. 2025MAGNET: A Multi-agent Framework for Finding Audio-Visual Needles by Reasoning over Multi-Video Haystackscited by 11
  38. 2025E-VRAG: Enhancing Long Video Understanding with Resource-Efficient Retrieval Augmented Generationcited by 11

50 most cited of 100 All 100 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025LLaVA-Mini: Efficient Image and Video Large Multimodal Models with One Vision Token
  2. 2025Multiscale Byte Language Models -- A Hierarchical Architecture for Causal Million-Length Sequence Modeling
  3. 2024MA-LMM: Memory-Augmented Large Multimodal Model for Long-Term Video Understanding
  4. 2026ParVL: Parallel Scaling and Expandable Compute Allocation for Multimodal LLMs
  5. 2024A Review of Multi-Modal Large Language and Vision Models
  6. 2023X-LLM: Bootstrapping Advanced Large Language Models by Treating Multi-Modalities as Foreign Languages
  7. 2026Cost-Efficient Multimodal LLM Inference via Cross-Tier GPU Heterogeneity
  8. 2024MammothModa: Multi-Modal Large Language Model