# MiniVLM: A Smaller and Faster Vision-Language Model

arXiv: https://arxiv.org/abs/2012.06946 · HTML: https://mlsanity.com/paper/2012.06946/

## Key facts

- Cited by 51 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 13 Dec 2020.

## Built on by (most cited first, 30 of 51)

- [Florence: A New Foundation Model for Computer Vision](https://mlsanity.com/paper/2111.11432/) (2021)
- [GIT: A Generative Image-to-text Transformer for Vision and Language](https://mlsanity.com/paper/2205.14100/) (2022)
- [Scaling Up Vision-Language Pre-training for Image Captioning](https://mlsanity.com/paper/2111.12233/) (2021)
- [Vision-Language Pre-training: Basics, Recent Advances, and Future Trends](https://mlsanity.com/paper/2210.09263/) (2022)
- [PromptCap: Prompt-Guided Task-Aware Image Captioning](https://mlsanity.com/paper/2211.09699/) (2022)
- [Compressing Visual-linguistic Model via Knowledge Distillation](https://mlsanity.com/paper/2104.02096/) (2021)
- [Tag2Text: Guiding Vision-Language Model via Image Tagging](https://mlsanity.com/paper/2303.05657/) (2023)
- [EfficientVLM: Fast and Accurate Vision-Language Models via Knowledge Distillation and Modal-adaptive Pruning](https://mlsanity.com/paper/2210.07795/) (2022)
- [UPop: Unified and Progressive Pruning for Compressing Vision-Language Transformers](https://mlsanity.com/paper/2301.13741/) (2023)
- [PuMer: Pruning and Merging Tokens for Efficient Vision Language Models](https://mlsanity.com/paper/2305.17530/) (2023)
- [UFO: A UniFied TransfOrmer for Vision-Language Representation Learning](https://mlsanity.com/paper/2111.10023/) (2021)
- [Playing Lottery Tickets with Vision and Language](https://mlsanity.com/paper/2104.11832/) (2021)
- [CrossGET: Cross-Guided Ensemble of Tokens for Accelerating Vision-Language Transformers](https://mlsanity.com/paper/2305.17455/) (2023)
- [Injecting Semantic Concepts into End-to-End Image Captioning](https://mlsanity.com/paper/2112.05230/) (2021)
- [Vision-Language Models for Edge Networks: A Comprehensive Survey](https://mlsanity.com/paper/2502.07855/) (2025)
- [Distilled Dual-Encoder Model for Vision-Language Understanding](https://mlsanity.com/paper/2112.08723/) (2021)
- [A Survey on Efficient Vision-Language Models](https://mlsanity.com/paper/2504.09724/) (2025)
- [TRIPS: Efficient Vision-and-Language Pre-training with Text-Relevant Image Patch Selection](https://mlsanity.com/paper/2305.04474/) (2023)
- [TrafficVLM: A Controllable Visual Language Model for Traffic Video Captioning](https://mlsanity.com/paper/2404.09275/) (2024)
- [Exploring Discrete Diffusion Models for Image Captioning](https://mlsanity.com/paper/2211.11694/) (2022)
- [Towards Models that Can See and Read](https://mlsanity.com/paper/2301.07389/) (2023)
- [Perspectives and Prospects on Transformer Architecture for Cross-Modal Tasks with Language and Vision](https://mlsanity.com/paper/2103.04037/) (2021)
- [Efficient Image Captioning for Edge Devices](https://mlsanity.com/paper/2212.08985/) (2022)
- [A Frustratingly Simple Approach for End-to-End Image Captioning](https://mlsanity.com/paper/2201.12723/) (2022)
- [Module-wise Adaptive Distillation for Multimodality Foundation Models](https://mlsanity.com/paper/2310.04550/) (2023)
- [Vision-Language Model for Object Detection and Segmentation: A Review and Evaluation](https://mlsanity.com/paper/2504.09480/) (2025)
- [LoopITR: Combining Dual and Cross Encoder Architectures for Image-Text Retrieval](https://mlsanity.com/paper/2203.05465/) (2022)
- [What Vision-Language Models `See' when they See Scenes](https://mlsanity.com/paper/2109.07301/) (2021)
- [SmartTrim: Adaptive Tokens and Attention Pruning for Efficient Vision-Language Models](https://mlsanity.com/paper/2305.15033/) (2023)
- [Several categories of Large Language Models (LLMs): A Short Survey](https://mlsanity.com/paper/2307.10188/) (2023)

## Abstract (arXiv)

Recent vision-language (VL) studies have shown remarkable progress by learning generic representations from massive image-text pairs with transformer models and then fine-tuning on downstream VL tasks. While existing research has been focused on achieving high accuracy with large pre-trained models, building a lightweight model is of great value in practice but is less explored. In this paper, we propose a smaller and faster VL model, MiniVLM, which can be finetuned with good performance on various downstream tasks like its larger counterpart. MiniVLM consists of two modules, a vision feature extractor and a transformer-based vision-language fusion module. We design a Two-stage Efficient feature Extractor (TEE), inspired by the one-stage EfficientDet network, to significantly reduce the time cost of visual feature extraction by 95%, compared to a baseline model. We adopt the MiniLM structure to reduce the computation cost of the transformer module after comparing different compact BERT models. In addition, we improve the MiniVLM pre-training by adding 7M Open Images data, which are pseudo-labeled by a state-of-the-art captioning model. We also pre-train with high-quality image tags obtained from a strong tagging model to enhance cross-modality alignment. The large models are used offline without adding any overhead in fine-tuning and inference. With the above design choices, our MiniVLM reduces the model size by 73% and the inference time cost by 94% while being able to retain 94-97% of the accuracy on multiple VL tasks. We hope that MiniVLM helps ease the use of the state-of-the-art VL research for on-the-edge applications.