mlsanityData to 14 Oct 2026

OmniVLM: A Token-Compressed, Sub-Billion-Parameter Vision-Language Model for Efficient On-Device Inference

arXiv 2412.11475 ↗First posted 16 Dec 2024cs.CVcited by 8

On the map: Multimodal LLMs / vision-language models → Visual token pruning and efficiency

Abstract (arXiv)

We present OmniVLM, a sub-billion-parameter vision-language model for efficient on-device inference. OmniVLM introduces a token compression mechanism that reduces visual token sequence length from 729 to 81 tokens, significantly reducing computational overhead while preserving visual-semantic fidelity. Through a multi-stage training pipeline of pretraining, supervised fine-tuning, and minimal-edit Direct Preference Optimization (DPO), OmniVLM matches the performance of larger models. On multiple benchmarks including ScienceQA, POPE, and MMMU, OmniVLM outperforms existing baselines like nanoLLAVA within a 968M-parameter footprint. Empirical results on the same laptop demonstrate 9.1x faster time-to-first-token (0.75s vs 6.82s) and 1.5x higher decoding speed (29.41 vs 19.20 tokens/s) compared to nanoLLAVA, enabling efficient deployment on edge devices. The model weights can be accessed on huggingface: https://huggingface.co/NexaAIDev/OmniVLM-968M, and the inference examples can be find in Appendix B.

Key facts. Cited by 8 papers in the map (data through 14 Oct 2026). Placed in Visual token pruning and efficiency, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 19 of its 27 references that are in the map. First posted on arXiv 16 Dec 2024.

Builds on

19 of 27 references.

References · 19

  1. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  2. 2024The Llama 3 Herd of Modelscited by 14,715
  3. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  4. 2023Visual Instruction Tuningcited by 7,869
  5. 2023Improved Baselines with Visual Instruction Tuningcited by 4,418
  6. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,234
  7. 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,109
  8. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Taskscited by 2,565
  9. 2023Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyondcited by 2,224
  10. 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answeringcited by 1,919
  11. 2024PaliGemma: A versatile 3B VLM for transfercited by 696
  12. 2023AppAgent: Multimodal Agents as Smartphone Userscited by 415
7 more
  1. 2023Efficient and Effective Text Encoding for Chinese LLaMA and Alpacacited by 305
  2. 2024MobileLLM: Optimizing Sub-billion Parameter Language Models for On-Device Use Casescited by 223
  3. 2024ALLaVA: Harnessing GPT4V-Synthesized Data for Lite Vision-Language Modelscited by 205
  4. 2024Ferret-UI: Grounded Mobile UI Understanding with Multimodal LLMscited by 176
  5. 2024Ferret-UI 2: Mastering Universal User Interface Understanding Across Platformscited by 51
  6. 2024Octopus v2: On-device language model for super agentcited by 47
  7. 2024Squid: Long Context as a New Modality for Energy-Efficient On-Device Language Modelscited by 6

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 7; 2026: 1.

8 papers.

  1. 2025AutoNeural: Co-Designing Vision-Language Models for NPU Inferencecited by 17
  2. 2025Cognitive Edge Computing: A Comprehensive Survey on Optimizing Large Models and AI Agents for Pervasive Deploymentcited by 10
  3. 2025Towards a Multi-Agent Vision-Language System for Zero-Shot Novel Hazardous Object Detection for Autonomous Driving Safetycited by 9
  4. 2025Masking Teacher and Reinforcing Student for Distilling Vision-Language Modelscited by 6
  5. 2025A Review of 3D Object Detection with Vision-Language Modelscited by 4
  6. 2025Towards Pervasive Distributed Agentic Generative AI -- A State of The Artcited by 4
  7. 2025Eye Gaze Tells You Where to Compute: Gaze-Driven Efficient VLMscited by 3
  8. 2026Risk-Adaptive Edge--Cloud Visual Reasoning for Communication-Efficient Autonomous Drivingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026CoVSpec: Efficient Device-Edge Co-Inference for Vision-Language Models via Speculative Decoding
  2. 2026Efficient Inference for Large Vision-Language Models: Bottlenecks, Techniques, and Prospects
  3. 2026Towards Efficient Large Vision-Language Models: A Comprehensive Survey on Inference Strategies
  4. 2026CIVIC: End-to-End Sequence Compactness for Efficient Vision-Language Models
  5. 2026SAGE: Efficient Protected Vision-Language Inference on Smart Glasses
  6. 2026VisionPsy-Nano: Improving Accuracy, Efficiency, and Reliability in On-Device Vision-Language Models
  7. 2026Empirical Recipes for Efficient and Compact Vision-Language Models
  8. 2025LiteVLM: A Low-Latency Vision-Language Model Inference Pipeline for Resource-Constrained Environments