mlsanityData to 7 Oct 2026

Empirical Recipes for Efficient and Compact Vision-Language Models

arXiv 2603.16987 ↗First posted 17 Mar 2026cs.CV cs.AIcited by 0

On the map: Multimodal LLMs / vision-language models → Visual token pruning and efficiency

also in 1 more
Abstract (arXiv)

Deploying vision-language models (VLMs) in resource-constrained settings demands low latency and high throughput, yet existing compact VLMs often fall short of the inference speedups their smaller parameter counts suggest. To explain this discrepancy, we conduct an empirical end-to-end efficiency analysis and systematically profile inference to identify the dominant bottlenecks. Based on these findings, we develop optimization recipes tailored to compact VLMs that substantially reduce latency while preserving accuracy. These techniques cut time to first token (TTFT) by 53% on InternVL3-2B and by 93% on SmolVLM-256M. Our recipes are broadly applicable across both VLM architectures and common serving frameworks, providing practical guidance for building efficient VLM systems. Beyond efficiency, we study how to extend compact VLMs with structured perception outputs and introduce the resulting model family, ArgusVLM. Across diverse benchmarks, ArgusVLM achieves strong performance while maintaining a compact and efficient design.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Visual token pruning and efficiency, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 40 of its 54 references that are in the map. First posted on arXiv 17 Mar 2026.

Builds on

40 of 54 references.

Closest ancestors · 3

Same direction, newest first

  1. 2025SmolVLM: Redefining small and efficient multimodal models7 Apr
  2. 2025LLaVA-Mini: Efficient Image and Video Large Multimodal Models with One Vision Token7 Jan
  3. 2023MobileVLM : A Fast, Strong and Open Vision Language Assistant for Mobile Devices28 Dec

Other references · 37

  1. 2017Attention Is All You Needcited by 59,932
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  5. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  6. 2025Qwen3 Technical Reportcited by 8,195
  7. 2023Visual Instruction Tuningcited by 7,792
  8. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  9. 2025Qwen2.5-VL Technical Reportcited by 5,418
  10. 2024Qwen2.5 Technical Reportcited by 4,713
  11. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
  12. 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,048
25 more
  1. 2024LLaVA-OneVision: Easy Visual Task Transfercited by 2,669
  2. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Taskscited by 2,534
  3. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,498
  4. 2019HuggingFace's Transformers: State-of-the-art Natural Language Processingcited by 2,201
  5. 2025InternVL3: Exploring Advanced Training and Test-Time Recipes for Open-Source Multimodal Modelscited by 1,769
  6. 2023Evaluating Object Hallucination in Large Vision-Language Modelscited by 1,696
  7. 2024Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scalingcited by 1,573
  8. 2016Real-Time Single Image and Video Super-Resolution Using an Efficient Sub-Pixel Convolutional Neural Networkcited by 1,199
  9. 2023ShareGPT4V: Improving Large Multi-Modal Models with Better Captionscited by 1,000
  10. 2023EVA-CLIP: Improved Training Techniques for CLIP at Scalecited by 786
  11. 2024DeepSeek-VL: Towards Real-World Vision-Language Understandingcited by 718
  12. 2018nocaps: novel object captioning at scalecited by 444
  13. 2022Unified-IO: A Unified Model for Vision, Language, and Multi-Modal Taskscited by 420
  14. 2023Florence-2: Advancing a Unified Representation for a Variety of Vision Taskscited by 403
  15. 2024InternLM2 Technical Reportcited by 391
  16. 2025Kimi-VL Technical Reportcited by 381
  17. 2024Mini-Gemini: Mining the Potential of Multi-modality Vision Language Modelscited by 361
  18. 2025SmolLM2: When Smol Goes Big -- Data-Centric Training of a Small Language Modelcited by 307
  19. 2023Unified-IO 2: Scaling Autoregressive Multimodal Models with Vision, Language, Audio, and Actioncited by 301
  20. 2015DenseCap: Fully Convolutional Localization Networks for Dense Captioningcited by 176
  21. 2024MobileVLM V2: Faster and Stronger Baseline for Vision Language Modelcited by 172
  22. 2023UReader: Universal OCR-free Visually-situated Language Understanding with Multimodal Large Language Modelcited by 143
  23. 2022GRiT: A Generative Region-to-text Transformer for Object Understandingcited by 119
  24. 2013Optimizing CUDA Code By Kernel Fusion---Application on BLAScited by 7
  25. 2024H2OVL-Mississippi Vision Language Models Technical Reportcited by 7

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025A Survey on Efficient Vision-Language Models
  2. 2025LiteVLM: A Low-Latency Vision-Language Model Inference Pipeline for Resource-Constrained Environments
  3. 2025Bridging vision language model (VLM) evaluation gaps with a framework for scalable and cost-effective benchmark generation
  4. 2024What matters when building vision-language models?
  5. 2025Efficient Test-Time Scaling for Small Vision-Language Models
  6. 2025EffiVLM-BENCH: A Comprehensive Benchmark for Evaluating Training-Free Acceleration in Large Vision-Language Models
  7. 2026Not All Error Yields to Scale: Where Scaling Stops in Vision-Language Inference
  8. 2024Inference Optimal VLMs Need Fewer Visual Tokens and More Parameters