# OmniVLM: A Token-Compressed, Sub-Billion-Parameter Vision-Language Model for Efficient On-Device Inference

arXiv: https://arxiv.org/abs/2412.11475 · HTML: https://mlsanity.com/paper/2412.11475/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Placed in Visual token pruning and efficiency, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 19 of its 27 references that are in the map.
- First posted on arXiv 16 Dec 2024.

## Placed in

- Multimodal LLMs / vision-language models / [Visual token pruning and efficiency](https://mlsanity.com/mllm/visual-token-pruning-and-efficiency/)

## Builds on (most cited first, 19 of 19)

- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://mlsanity.com/paper/2306.05685/) (2023)
- [Visual Instruction Tuning](https://mlsanity.com/paper/2304.08485/) (2023)
- [Improved Baselines with Visual Instruction Tuning](https://mlsanity.com/paper/2310.03744/) (2023)
- [Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolution](https://mlsanity.com/paper/2409.12191/) (2024)
- [Sigmoid Loss for Language Image Pre-Training](https://mlsanity.com/paper/2303.15343/) (2023)
- [InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Tasks](https://mlsanity.com/paper/2312.14238/) (2023)
- [Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyond](https://mlsanity.com/paper/2308.12966/) (2023)
- [Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answering](https://mlsanity.com/paper/2209.09513/) (2022)
- [PaliGemma: A versatile 3B VLM for transfer](https://mlsanity.com/paper/2407.07726/) (2024)
- [AppAgent: Multimodal Agents as Smartphone Users](https://mlsanity.com/paper/2312.13771/) (2023)
- [Efficient and Effective Text Encoding for Chinese LLaMA and Alpaca](https://mlsanity.com/paper/2304.08177/) (2023)
- [MobileLLM: Optimizing Sub-billion Parameter Language Models for On-Device Use Cases](https://mlsanity.com/paper/2402.14905/) (2024)
- [ALLaVA: Harnessing GPT4V-Synthesized Data for Lite Vision-Language Models](https://mlsanity.com/paper/2402.11684/) (2024)
- [Ferret-UI: Grounded Mobile UI Understanding with Multimodal LLMs](https://mlsanity.com/paper/2404.05719/) (2024)
- [Ferret-UI 2: Mastering Universal User Interface Understanding Across Platforms](https://mlsanity.com/paper/2410.18967/) (2024)
- [Octopus v2: On-device language model for super agent](https://mlsanity.com/paper/2404.01744/) (2024)
- [Squid: Long Context as a New Modality for Energy-Efficient On-Device Language Models](https://mlsanity.com/paper/2408.15518/) (2024)

## Built on by (most cited first, 8 of 8)

- [AutoNeural: Co-Designing Vision-Language Models for NPU Inference](https://mlsanity.com/paper/2512.02924/) (2025)
- [Cognitive Edge Computing: A Comprehensive Survey on Optimizing Large Models and AI Agents for Pervasive Deployment](https://mlsanity.com/paper/2501.03265/) (2025)
- [Towards a Multi-Agent Vision-Language System for Zero-Shot Novel Hazardous Object Detection for Autonomous Driving Safety](https://mlsanity.com/paper/2504.13399/) (2025)
- [Masking Teacher and Reinforcing Student for Distilling Vision-Language Models](https://mlsanity.com/paper/2512.22238/) (2025)
- [A Review of 3D Object Detection with Vision-Language Models](https://mlsanity.com/paper/2504.18738/) (2025)
- [Towards Pervasive Distributed Agentic Generative AI -- A State of The Art](https://mlsanity.com/paper/2506.13324/) (2025)
- [Eye Gaze Tells You Where to Compute: Gaze-Driven Efficient VLMs](https://mlsanity.com/paper/2509.16476/) (2025)
- [Risk-Adaptive Edge--Cloud Visual Reasoning for Communication-Efficient Autonomous Driving](https://mlsanity.com/paper/2608.14991/) (2026)

## Abstract (arXiv)

We present OmniVLM, a sub-billion-parameter vision-language model for efficient on-device inference. OmniVLM introduces a token compression mechanism that reduces visual token sequence length from 729 to 81 tokens, significantly reducing computational overhead while preserving visual-semantic fidelity. Through a multi-stage training pipeline of pretraining, supervised fine-tuning, and minimal-edit Direct Preference Optimization (DPO), OmniVLM matches the performance of larger models. On multiple benchmarks including ScienceQA, POPE, and MMMU, OmniVLM outperforms existing baselines like nanoLLAVA within a 968M-parameter footprint. Empirical results on the same laptop demonstrate 9.1x faster time-to-first-token (0.75s vs 6.82s) and 1.5x higher decoding speed (29.41 vs 19.20 tokens/s) compared to nanoLLAVA, enabling efficient deployment on edge devices. The model weights can be accessed on huggingface: https://huggingface.co/NexaAIDev/OmniVLM-968M, and the inference examples can be find in Appendix B.