mlsanityData to 7 Oct 2026

NVILA: Efficient Frontier Visual Language Models

arXiv 2412.04468 ↗First posted 5 Dec 2024cs.CVcited by 206

On the map: Multimodal LLMs / vision-language models → VLM benchmarks and visual reasoning

Abstract (arXiv)

Visual language models (VLMs) have made significant advances in accuracy in recent years. However, their efficiency has received much less attention. This paper introduces NVILA, a family of open VLMs designed to jointly optimize efficiency and accuracy. Building on top of VILA, we improve its model architecture by first scaling up the spatial and temporal resolutions, and then compressing visual tokens. This "scale-then-compress" approach enables NVILA to efficiently process high-resolution images and long videos. We further conduct a systematic investigation that enhances NVILA's efficiency throughout its entire lifecycle, from training and fine-tuning to deployment. NVILA matches or surpasses the accuracy of leading open and proprietary VLMs across a wide range of image and video benchmarks. At the same time, it reduces training cost by 1.9-5.1x, prefilling latency by 1.6-2.2x, and decoding latency by 1.2-2.8x. We release our code and models to facilitate reproducibility.

Key facts. Cited by 206 papers in the map (data through 7 Oct 2026). Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 123 of its 144 references that are in the map. First posted on arXiv 5 Dec 2024.

Builds on

123 of 144 references.

Closest ancestors · 5

Same direction, newest first

  1. 2024Building and better understanding vision-language models: insights and future directions22 Aug
  2. 2024VILA2: VILA Augmented VILA24 Jul
  3. 2024What matters when building vision-language models?3 May
  4. 2024ALLaVA: Harnessing GPT4V-Synthesized Data for Lite Vision-Language Models18 Feb
  5. 2023MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contexts3 Oct

Other references · 118

  1. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  2. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  3. 2023Visual Instruction Tuningcited by 7,792
  4. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  5. 2019nuScenes: A multimodal dataset for autonomous drivingcited by 4,485
  6. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,151
  7. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
  8. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,355
  9. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,175
  10. 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,048
  11. 2024LLaVA-OneVision: Easy Visual Task Transfercited by 2,669
  12. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,551
38 more
  1. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Taskscited by 2,534
  2. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,498
  3. 2015VQA: Visual Question Answeringcited by 2,489
  4. 2024Qwen2 Technical Reportcited by 2,076
  5. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,044
  6. 2022RT-1: Robotics Transformer for Real-World Control at Scalecited by 2,007
  7. 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answeringcited by 1,881
  8. 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,838
  9. 2019Towards VQA Models That Can Readcited by 1,696
  10. 2022ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoningcited by 1,449
  11. 2016CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoningcited by 1,445
  12. 2024Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysiscited by 1,375
  13. 2020DocVQA: A Dataset for VQA on Document Imagescited by 1,288
  14. 2023AWQ: Activation-aware Weight Quantization for LLM Compression and Accelerationcited by 1,234
  15. 2017Mixed Precision Trainingcited by 1,135
  16. 2019OK-VQA: A Visual Question Answering Benchmark Requiring External Knowledgecited by 1,114
  17. 2023MVBench: A Comprehensive Multi-modal Video Understanding Benchmarkcited by 1,086
  18. 2024MiniCPM-V: A GPT-4V Level MLLM on Your Phonecited by 1,061
  19. 2016Modeling Context in Referring Expressionscited by 1,054
  20. 2024How Far Are We to GPT-4V? Closing the Gap to Commercial Multimodal Models with Open-Source Suitescited by 1,033
  21. 2017Vision-and-Language Navigation: Interpreting visually-grounded navigation instructions in real environmentscited by 1,013
  22. 2023ShareGPT4V: Improving Large Multi-Modal Models with Better Captionscited by 1,000
  23. 2022Token Merging: Your ViT But Fastercited by 995
  24. 2023SEED-Bench: Benchmarking Multimodal LLMs with Generative Comprehensioncited by 929
  25. 2016Temporal Segment Networks: Towards Good Practices for Deep Action Recognitioncited by 899
  26. 2023Shikra: Unleashing Multimodal LLM's Referential Dialogue Magiccited by 891
  27. 2024Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMscited by 822
  28. 2016A Diagram Is Worth A Dozen Imagescited by 793
  29. 2019ALFRED: A Benchmark for Interpreting Grounded Instructions for Everyday Taskscited by 773
  30. 2024DoRA: Weight-Decomposed Low-Rank Adaptationcited by 740
  31. 2023VILA: On Pre-training for Visual Language Modelscited by 686
  32. 2024An Image is Worth 1/2 Tokens After Layer 2: Plug-and-Play Inference Acceleration for Large Vision-Language Modelscited by 629
  33. 2017Towards Automatic Learning of Procedures from Web Instructional Videoscited by 616
  34. 2019ActivityNet-QA: A Dataset for Understanding Complex Web Videos via Question Answeringcited by 559
  35. 2024LLaVA-Video: Video Instruction Tuning With Synthetic Datacited by 550
  36. 2024LESS: Selecting Influential Data for Targeted Instruction Tuningcited by 514
  37. 2023MAmmoTH: Building Math Generalist Models through Hybrid Instruction Tuningcited by 483
  38. 2024DriveVLM: The Convergence of Autonomous Driving and Large Vision-Language Modelscited by 482

50 most cited of 118 All 118 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 4; 2025: 123; 2026: 79.

206 papers.

Sort:

By area:

  1. 2025V-JEPA 2: Self-Supervised Video Models Enable Understanding, Prediction and Planningcited by 738
  2. 2025VideoLLaMA 3: Frontier Multimodal Foundation Models for Image and Video Understandingcited by 502
  3. 2025SmolVLM: Redefining small and efficient multimodal modelscited by 226
  4. 2024OCRBench v2: An Improved Benchmark for Evaluating Large Multimodal Models on Visual Text Localization and Reasoningcited by 165
  5. 2025SANA 1.5: Efficient Scaling of Training-Time and Inference-Time Compute in Linear Diffusion Transformercited by 165
  6. 2025ThinkAct: Vision-Language-Action Reasoning via Reinforced Visual Latent Planningcited by 152
  7. 2025PhysBench: Benchmarking and Enhancing Vision-Language Models for Physical World Understandingcited by 147
  8. 2025WorldSense: Evaluating Real-world Omnimodal Understanding for Multimodal LLMscited by 147
  9. 2025LlamaV-o1: Rethinking Step-by-step Visual Reasoning in LLMscited by 145
  10. 2025Daily-Omni: Towards Audio-Visual Reasoning with Temporal Alignment across Modalitiescited by 101
  11. 2025Scaling RL to Long Videoscited by 83
  12. 2025LaViDa: A Large Diffusion Language Model for Multimodal Understandingcited by 81
38 more
  1. 2025StreamForest: Efficient Online Video Understanding with Persistent Event Memorycited by 71
  2. 2025Eagle 2: Building Post-Training Data Strategies from Scratch for Frontier Vision-Language Modelscited by 69
  3. 2025Eagle 2.5: Boosting Long-Context Post-Training for Frontier Vision-Language Modelscited by 62
  4. 2025HoliTom: Holistic Token Merging for Fast Video Large Language Modelscited by 58
  5. 2025StreamBridge: Turning Your Offline Video Large Language Model into a Proactive Streaming Assistantcited by 56
  6. 2025EmbodiedEval: Evaluate Multimodal LLMs as Embodied Agentscited by 55
  7. 2025OmniVinci: Enhancing Architecture and Data for Omni-Modal Understanding LLMcited by 53
  8. 2024FrameFusion: Combining Similarity and Importance for Video Token Reduction on Large Vision Language Modelscited by 52
  9. 2025FUDOKI: Discrete Flow-based Unified Understanding and Generation via Kinetic-Optimal Velocitiescited by 46
  10. 2025Long-VITA: Scaling Large Multi-modal Models to 1 Million Tokens with Leading Short-Context Accuracycited by 39
  11. 2025video-SALMONN 2: Caption-Enhanced Audio-Visual Large Language Modelscited by 39
  12. 2025Video-XL-2: Towards Very Long-Video Understanding Through Task-Aware KV Sparsificationcited by 36
  13. 2025Generative Frame Sampler for Long Video Understandingcited by 35
  14. 2025UniGen: Enhanced Training & Test-Time Strategies for Unified Multimodal Understanding and Generationcited by 32
  15. 2025FOLDER: Accelerating Multi-modal Large Language Models with Enhanced Performancecited by 31
  16. 2025Mavors: Multi-granularity Video Representation for Multimodal Large Language Modelcited by 31
  17. 20253D Aware Region Prompted Vision Language Modelcited by 31
  18. 2026LLaVA-OneVision-2: Towards Next-Generation Perceptual Intelligencecited by 31
  19. 2025SlowFast-LLaVA-1.5: A Family of Token-Efficient Video Large Language Models for Long-Form Video Understandingcited by 30
  20. 2025MMLongBench: Benchmarking Long-Context Vision-Language Models Effectively and Thoroughlycited by 29
  21. 2025VideoExplorer: Think With Videos For Agentic Long-Video Understandingcited by 29
  22. 2025LLaVA-MORE: A Comparative Study of LLMs and Visual Backbones for Enhanced Visual Instruction Tuningcited by 27
  23. 2025TIGeR: Tool-Integrated Geometric Reasoning in Vision-Language Models for Roboticscited by 27
  24. 2025Improving LLM Video Understanding with 16 Frames Per Secondcited by 26
  25. 2025Scaling Vision Pre-Training to 4K Resolutioncited by 26
  26. 2024StreamChat: Chatting with Streaming Videocited by 23
  27. 2025CAPability: A Comprehensive Visual Caption Benchmark for Evaluating Both Correctness and Thoroughnesscited by 23
  28. 2025MR. Video: "MapReduce" is the Principle for Long Video Understandingcited by 23
  29. 2025Do Vision-Language Models Have Internal World Models? Towards an Atomic Evaluationcited by 23
  30. 2025TSPO: Temporal Sampling Policy Optimization for Long-form Video Language Understandingcited by 23
  31. 2025MMR-V: What's Left Unsaid? A Benchmark for Multimodal Deep Reasoning in Videoscited by 22
  32. 2025EgoExoBench: A Benchmark for First- and Third-person View Video Understanding in MLLMscited by 22
  33. 2025Visual Attention Never Fades: Selective Progressive Attention ReCalibration for Detailed Image Captioning in Multimodal Large Language Modelscited by 21
  34. 2025Referring to Any Personcited by 20
  35. 2025EOC-Bench: Can MLLMs Identify, Recall, and Forecast Objects in an Egocentric World?cited by 20
  36. 2026Fast-ThinkAct: Efficient Vision-Language-Action Reasoning via Verbalizable Latent Planningcited by 20
  37. 2025VideoMem: Enhancing Ultra-Long Video Understanding via Adaptive Memory Managementcited by 19
  38. 2025MME-VideoOCR: Evaluating OCR-Based Capabilities of Multimodal LLMs in Video Scenarioscited by 18

50 most cited of 206 All 206 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Visual Large Language Models for Generalized and Specialized Applications
  2. 2025Native Visual Understanding: Resolving Resolution Dilemmas in Vision-Language Models
  3. 2024An Introduction to Vision-Language Modeling
  4. 2025NaViL: Rethinking Scaling Properties of Native Multimodal Large Language Models under Data Constraints
  5. 202620/20 Vision Language Models: A Prescription for Better VLMs through Data Curation Alone
  6. 2021SimVLM: Simple Visual Language Model Pretraining with Weak Supervision
  7. 2025AdaFV: Rethinking of Visual-Language alignment for VLM acceleration
  8. 2025EffiVLM-BENCH: A Comprehensive Benchmark for Evaluating Training-Free Acceleration in Large Vision-Language Models