mlsanityData to 7 Oct 2026

SmolVLM: Redefining small and efficient multimodal models

arXiv 2504.05299 ↗First posted 7 Apr 2025cs.AI cs.CVcited by 226

On the map: Multimodal LLMs / vision-language models → Visual token pruning and efficiency

Abstract (arXiv)

Large Vision-Language Models (VLMs) deliver exceptional performance but require significant computational resources, limiting their deployment on mobile and edge devices. Smaller VLMs typically mirror design choices of larger models, such as extensive image tokenization, leading to inefficient GPU memory usage and constrained practicality for on-device applications. We introduce SmolVLM, a series of compact multimodal models specifically engineered for resource-efficient inference. We systematically explore architectural configurations, tokenization strategies, and data curation optimized for low computational overhead. Through this, we identify key design choices that yield substantial performance gains on image and video tasks with minimal memory footprints. Our smallest model, SmolVLM-256M, uses less than 1GB GPU memory during inference and outperforms the 300-times larger Idefics-80B model, despite an 18-month development gap. Our largest model, at 2.2B parameters, rivals state-of-the-art VLMs consuming twice the GPU memory. SmolVLM models extend beyond static images, demonstrating robust video comprehension capabilities. Our results emphasize that strategic architectural optimizations, aggressive yet efficient tokenization, and carefully curated training data significantly enhance multimodal performance, facilitating practical, energy-efficient deployments at significantly smaller scales.

Key facts. Cited by 226 papers in the map (data through 7 Oct 2026). Placed in Visual token pruning and efficiency, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 72 of its 83 references that are in the map. First posted on arXiv 7 Apr 2025.

Builds on

72 of 83 references.

References · 72

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2023GPT-4 Technical Reportcited by 18,132
  3. 2024The Llama 3 Herd of Modelscited by 14,192
  4. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  5. 2023Visual Instruction Tuningcited by 7,792
  6. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 6,146
  7. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  8. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  9. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,151
  10. 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,048
  11. 2022Training Compute-Optimal Large Language Modelscited by 2,927
  12. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,498
38 more
  1. 2015VQA: Visual Question Answeringcited by 2,489
  2. 2023Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyondcited by 2,179
  3. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,044
  4. 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answeringcited by 1,881
  5. 2019Towards VQA Models That Can Readcited by 1,696
  6. 2023MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contextscited by 1,582
  7. 2024Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scalingcited by 1,573
  8. 2022ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoningcited by 1,449
  9. 2023Video-LLaVA: Learning United Visual Representation by Alignment Before Projectioncited by 1,385
  10. 2024Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysiscited by 1,375
  11. 2020DocVQA: A Dataset for VQA on Document Imagescited by 1,288
  12. 2016Real-Time Single Image and Video Super-Resolution Using an Efficient Sub-Pixel Convolutional Neural Networkcited by 1,199
  13. 2021Perceiver: General Perception with Iterative Attentioncited by 1,089
  14. 2023MVBench: A Comprehensive Multi-modal Video Understanding Benchmarkcited by 1,086
  15. 2024MiniCPM-V: A GPT-4V Level MLLM on Your Phonecited by 1,061
  16. 2023Video-ChatGPT: Towards Detailed Video Understanding via Large Vision and Language Modelscited by 1,047
  17. 2024How Far Are We to GPT-4V? Closing the Gap to Commercial Multimodal Models with Open-Source Suitescited by 1,033
  18. 2023ShareGPT4V: Improving Large Multi-Modal Models with Better Captionscited by 1,000
  19. 2024Are We on the Right Way for Evaluating Large Vision-Language Models?cited by 893
  20. 2024Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMscited by 822
  21. 2016A Diagram Is Worth A Dozen Imagescited by 793
  22. 2024DeepSeek-VL: Towards Real-World Vision-Language Understandingcited by 718
  23. 2024MiniCPM: Unveiling the Potential of Small Language Models with Scalable Training Strategiescited by 694
  24. 2024PaliGemma: A versatile 3B VLM for transfercited by 679
  25. 2023Textbooks Are All You Need II: phi-1.5 technical reportcited by 559
  26. 2023MovieChat: From Dense Token to Sparse Memory for Long Video Understandingcited by 536
  27. 2025VideoLLaMA 3: Frontier Multimodal Foundation Models for Image and Video Understandingcited by 502
  28. 2023MAmmoTH: Building Math Generalist Models through Hybrid Instruction Tuningcited by 483
  29. 2024Dolma: an Open Corpus of Three Trillion Tokens for Language Model Pretraining Researchcited by 475
  30. 2023OCRBench: On the Hidden Mystery of OCR in Large Multimodal Modelscited by 466
  31. 2024VLMEvalKit: An Open-Source Toolkit for Evaluating Large Multi-Modality Modelscited by 443
  32. 2024DataComp-LM: In search of the next generation of training sets for language modelscited by 381
  33. 2022The Stack: 3 TB of permissively licensed source codecited by 354
  34. 2024TempCompass: Do Video LLMs Really Understand Videos?cited by 352
  35. 2024Magpie: Alignment Data Synthesis from Scratch by Prompting Aligned LLMs with Nothingcited by 341
  36. 2019Scene Text Visual Question Answeringcited by 327
  37. 2024VideoAgent: Long-form Video Understanding with Large Language Model as Agentcited by 323
  38. 2024What matters when building vision-language models?cited by 321

50 most cited of 72 All 72 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 93; 2026: 133.

226 papers.

Sort:

By area:

  1. 2025InternVL3: Exploring Advanced Training and Test-Time Recipes for Open-Source Multimodal Modelscited by 1,769
  2. 2025SmolVLA: A Vision-Language-Action Model for Affordable and Efficient Roboticscited by 507
  3. 2026GigaWorld-Policy: An Efficient Action-Centered World--Action Modelcited by 129
  4. 2025Cambrian-S: Towards Spatial Supersensing in Videocited by 123
  5. 2025VL-JEPA: Joint Embedding Predictive Architecture for Vision-languagecited by 61
  6. 2025FLOWER: Democratizing Generalist Robot Policies with Efficient Vision-Language-Action Flow Policiescited by 56
  7. 2025FineVision: Open Data Is All You Needcited by 51
  8. 2025GigaBrain-0: A World Model-Powered Vision-Language-Action Modelcited by 43
  9. 2025Evo-1: Lightweight Vision-Language-Action Model with Preserved Semantic Alignmentcited by 35
  10. 2025A Survey on Efficient Vision-Language-Action Modelscited by 34
  11. 2026DynamicVLA: A Vision-Language-Action Model for Dynamic Object Manipulationcited by 33
  12. 2025VideoHallu: Evaluating and Mitigating Multi-modal Hallucinations on Synthetic Video Understandingcited by 32
38 more
  1. 2025MMTok: Multimodal Coverage Maximization for Efficient Inference of VLMscited by 29
  2. 2025Efficient Vision-Language-Action Models for Embodied Manipulation: A Systematic Surveycited by 25
  3. 2025SwiftVLA: Unlocking Spatiotemporal Dynamics for Lightweight VLA Models at Minimal Overheadcited by 23
  4. 2026GigaBrain-0.5M*: a VLA That Learns From World Model-Based Reinforcement Learningcited by 23
  5. 2026ViDoRe V3: A Comprehensive Evaluation of Retrieval Augmented Generation in Complex Real-World Scenarioscited by 22
  6. 2025Latent Action Pretraining Through World Modelingcited by 21
  7. 2025A Survey on Agentic Multimodal Large Language Modelscited by 19
  8. 2025UniversalRAG: Retrieval-Augmented Generation over Corpora of Diverse Modalities and Granularitiescited by 18
  9. 2025AutoNeural: Co-Designing Vision-Language Models for NPU Inferencecited by 17
  10. 2026ActionCodec: What Makes for Good Action Tokenizerscited by 17
  11. 2025Co-Reinforcement Learning for Unified Multimodal Understanding and Generationcited by 15
  12. 2025Turbo-VAED: Fast and Stable Transfer of Video-VAEs to Mobile Devicescited by 15
  13. 2025QSVD: Efficient Low-rank Approximation for Unified Query-Key-Value Weight Compression in Low-Precision Vision-Language Modelscited by 15
  14. 2025DREAM: Drafting with Refined Target Features and Entropy-Adaptive Cross-Attention Fusion for Multimodal Speculative Decodingcited by 13
  15. 2025DRISHTIKON: A Multimodal Multilingual Benchmark for Testing Language Models' Understanding on Indian Culturecited by 13
  16. 2025Understanding Language Prior of LVLMs by Contrasting Chain-of-Embeddingcited by 13
  17. 2025ELV-Halluc: Benchmarking Semantic Aggregation Hallucinations in Long Video Understandingcited by 10
  18. 2026Action100M: A Large-scale Video Action Datasetcited by 10
  19. 2026SimVLA: A Simple VLA Baseline for Robotic Manipulationcited by 10
  20. 2025Towards Understanding Visual Grounding in Visual Language Modelscited by 9
  21. 2025Lite VLA: Efficient Vision-Language-Action Control on CPU-Bound Edge Robotscited by 9
  22. 2025InfiniteVL: Synergizing Linear and Sparse Attention for Highly-Efficient, Unlimited-Input Vision-Language Modelscited by 9
  23. 2025OpenEvents V1: Large-Scale Benchmark Dataset for Multimodal Event Groundingcited by 8
  24. 2025CropVLM: Learning to Zoom for Fine-Grained Vision-Language Perceptioncited by 8
  25. 2026ResAdapt: Adaptive Resolution for Efficient Multimodal Reasoningcited by 8
  26. 2025LLM-Guided Agentic Object Detection for Open-World Understandingcited by 7
  27. 2025ReasonVQA: A Multi-hop Reasoning Benchmark with Structural Knowledge for Visual Question Answeringcited by 7
  28. 2025CaTS-Bench: Can Language Models Describe Time Series?cited by 7
  29. 2025ChartAgent: A Multimodal Agent for Visually Grounded Reasoning in Complex Chart Question Answeringcited by 7
  30. 2025Temporal Concept Dynamics in Diffusion Models via Prompt-Conditioned Interventionscited by 7
  31. 2026Mean-Flow based One-Step Vision-Language-Actioncited by 7
  32. 2026Penguin-VL: Exploring the Efficiency Limits of VLM with LLM-based Vision Encoderscited by 7
  33. 2025Test-Time Matching: Unlocking Compositional Reasoning in Multimodal Modelscited by 6
  34. 2025Masking Teacher and Reinforcing Student for Distilling Vision-Language Modelscited by 6
  35. 2025VL-RouterBench: A Benchmark for Vision-Language Model Routingcited by 6
  36. 2026Adaptive Capacity Allocation for Vision Language Action Fine-tuningcited by 6
  37. 2026LLMind: Bio-inspired Training-free Adaptive Visual Representations for Vision-Language Modelscited by 6
  38. 2026Orca: The World is in Your Mindcited by 6

50 most cited of 226 All 226 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026Towards Efficient Large Vision-Language Models: A Comprehensive Survey on Inference Strategies
  2. 2023MobileVLM : A Fast, Strong and Open Vision Language Assistant for Mobile Devices
  3. 2025Enabling Disaggregated Multi-Stage MLLM Inference via GPU-Internal Scheduling and Resource Sharing
  4. 2026Cost-Efficient Multimodal LLM Inference via Cross-Tier GPU Heterogeneity
  5. 2025Modality Inflation: Energy Characterization and Optimization Opportunities for MLLM Inference ↗
  6. 2026ParVL: Parallel Scaling and Expandable Compute Allocation for Multimodal LLMs
  7. 2025HyperVL: An Efficient and Dynamic Multimodal Large Language Model for Edge Devices
  8. 2026Efficient Inference for Large Vision-Language Models: Bottlenecks, Techniques, and Prospects