mlsanityData to 7 Oct 2026

EfficientVLM: Fast and Accurate Vision-Language Models via Knowledge Distillation and Modal-adaptive Pruning

arXiv 2210.07795 ↗First posted 14 Oct 2022cs.CL cs.CVcited by 56

On the map: Multimodal LLMs / vision-language models → Visual token pruning and efficiency

Abstract (arXiv)

Pre-trained vision-language models (VLMs) have achieved impressive results in a range of vision-language tasks. However, popular VLMs usually consist of hundreds of millions of parameters which brings challenges for fine-tuning and deployment in real-world applications due to space, memory, and latency constraints. In this work, we introduce a distilling then pruning framework to compress large vision-language models into smaller, faster, and more accurate ones. We first shrink the size of a pre-trained large VLM and apply knowledge distillation in the vision-language pre-training stage to obtain a task-agnostic compact VLM. Then we propose a modal-adaptive pruning algorithm to automatically infer the importance of vision and language modalities for different downstream tasks and adaptively remove redundant structures and neurons in different encoders with controllable target sparsity. We apply our framework to train EfficientVLM, a fast and accurate vision-language model consisting of 6 vision layers, 3 text layers, and 3 cross-modal fusion layers, accounting for only 93 million parameters in total, which is 44.3% of the teacher model. EfficientVLM retains 98.4% performance of the teacher model and accelerates its inference speed by 2.2x. EfficientVLM achieves a large absolute improvement over previous SoTA efficient VLMs of similar sizes by a large margin on various vision-language tasks, including VQAv2 (+4.9%), NLVR2 (+5.6%), ITR (R@1 on TR +17.2%, on IR + 15.6% ) and COCO caption generation (CIDEr +6.5), demonstrating a large potential on training lightweight VLMs.

Key facts. Cited by 56 papers in the map (data through 7 Oct 2026). Placed in Visual token pruning and efficiency, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 65 of its 83 references that are in the map. First posted on arXiv 14 Oct 2022.

Builds on

65 of 83 references.

References · 65

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  4. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  5. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  6. 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,617
  7. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,243
  8. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,646
  9. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,498
  10. 2018The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networkscited by 2,250
  11. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,219
  12. 2014CIDEr: Consensus-based Image Description Evaluationcited by 2,027
38 more
  1. 2021Align before Fuse: Vision and Language Representation Learning with Momentum Distillationcited by 1,612
  2. 2019LXMERT: Learning Cross-Modality Encoder Representations from Transformerscited by 1,548
  3. 2015Microsoft COCO Captions: Data Collection and Evaluation Servercited by 1,447
  4. 2020MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformerscited by 1,414
  5. 2017Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answeringcited by 1,310
  6. 2019VisualBERT: A Simple and Performant Baseline for Vision and Languagecited by 1,291
  7. 2021ViLT: Vision-and-Language Transformer Without Convolution or Region Supervisioncited by 1,224
  8. 2020Oscar: Object-Semantics Aligned Pre-training for Vision-Language Taskscited by 1,213
  9. 2019TinyBERT: Distilling BERT for Natural Language Understandingcited by 1,153
  10. 2016The Concrete Distribution: A Continuous Relaxation of Discrete Random Variablescited by 1,135
  11. 2014Deep Visual-Semantic Alignments for Generating Image Descriptionscited by 966
  12. 2019VL-BERT: Pre-training of Generic Visual-Linguistic Representationscited by 954
  13. 2016SPICE: Semantic Propositional Image Caption Evaluationcited by 820
  14. 2019Are Sixteen Heads Really Better than One?cited by 803
  15. 2019Green AIcited by 620
  16. 2017BranchyNet: Fast Inference via Early Exiting from Deep Neural Networkscited by 591
  17. 2017Learning Sparse Neural Networks through L0 Regularizationcited by 571
  18. 2019Patient Knowledge Distillation for BERT Model Compressioncited by 502
  19. 2016Adaptive Computation Time for Recurrent Neural Networkscited by 479
  20. 2018A Corpus for Reasoning About Natural Language Grounded in Photographscited by 457
  21. 2019Unicoder-VL: A Universal Encoder for Vision and Language by Cross-modal Pre-trainingcited by 451
  22. 2021VLMo: Unified Vision-Language Pre-Training with Mixture-of-Modality-Expertscited by 424
  23. 2020Movement Pruning: Adaptive Sparsity by Fine-Tuningcited by 398
  24. 2019Q-BERT: Hessian Based Ultra Low Precision Quantization of BERTcited by 355
  25. 2020Parameter-Efficient Transfer Learning with Diff Pruningcited by 351
  26. 2020Large-Scale Adversarial Training for Vision-and-Language Representation Learningcited by 327
  27. 2020DeeBERT: Dynamic Early Exiting for Accelerating BERT Inferencecited by 322
  28. 2019Q8BERT: Quantized 8Bit BERTcited by 296
  29. 2020Pixel-BERT: Aligning Image Pixels with Text by Deep Multi-Modal Transformerscited by 288
  30. 2020BERT Loses Patience: Fast and Robust Inference with Early Exitcited by 267
  31. 2021Multi-Grained Vision Language Pre-Training: Aligning Texts with Visual Conceptscited by 255
  32. 2020UNIMO: Towards Unified-Modal Understanding and Generation via Cross-Modal Contrastive Learningcited by 239
  33. 2020FastBERT: a Self-distilling BERT with Adaptive Inference Timecited by 224
  34. 2019Structured Pruning of Large Language Modelscited by 201
  35. 2021Seeing Out of tHe bOx: End-to-End Pre-training for Vision-Language Representation Learningcited by 187
  36. 2020Compressing BERT: Studying the Effects of Weight Pruning on Transfer Learningcited by 184
  37. 2021Block Pruning For Faster Transformerscited by 176
  38. 2022Structured Pruning Learns Compact and Accurate Modelscited by 167

50 most cited of 65 All 65 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 1; 2023: 11; 2024: 10; 2025: 24; 2026: 10.

56 papers.

Sort:

By area:

  1. 2023Vision-Language Foundation Models as Effective Robot Imitatorscited by 344
  2. 2025EfficientVLA: Training-Free Acceleration and Compression for Vision-Language-Action Modelscited by 89
  3. 2025MoLe-VLA: Dynamic Layer-skipping Vision Language Action Model via Mixture-of-Layers for Efficient Robot Manipulationcited by 69
  4. 2024V2X-VLM: End-to-End V2X Cooperative Autonomous Driving Through Large Vision-Language Modelscited by 47
  5. 2023CrossGET: Cross-Guided Ensemble of Tokens for Accelerating Vision-Language Transformerscited by 44
  6. 2025Efficient Agents: Building Effective Agents While Reducing Costcited by 32
  7. 2025FOLDER: Accelerating Multi-modal Large Language Models with Enhanced Performancecited by 31
  8. 2024MoPE-CLIP: Structured Pruning for Efficient Vision-Language Models with Module-wise Pruning Error Metriccited by 30
  9. 2025Vision-Language Models for Edge Networks: A Comprehensive Surveycited by 29
  10. 2024MIO: A Foundation Model on Multimodal Tokenscited by 24
  11. 2025AVA-VLA: Improving Vision-Language-Action models with Active Visual Attentioncited by 22
  12. 2024Align-KD: Distilling Cross-Modal Alignment Knowledge for Mobile Vision-Language Model Enhancementcited by 21
38 more
  1. 2024Active Data Curation Effectively Distills Large-Scale Multimodal Modelscited by 19
  2. 2023Knowledge Transfer from Vision Foundation Models for Efficient Training of Small Task-specific Modelscited by 14
  3. 2024MULTIFLOW: Shifting Towards Task-Agnostic Vision-Language Pruningcited by 13
  4. 2024Turbo: Informativity-Driven Acceleration Plug-In for Vision-Language Large Modelscited by 13
  5. 2023Turbo: Informativity-Driven Acceleration Plug-In for Vision-Language Modelscited by 11
  6. 2025UORA: Uniform Orthogonal Reinitialization Adaptation in Parameter-Efficient Fine-Tuning of Large Modelscited by 11
  7. 2024HoVLE: Unleashing the Power of Monolithic Vision-Language Models with Holistic Vision-Language Embeddingcited by 10
  8. 2025Vision-Language Model for Object Detection and Segmentation: A Review and Evaluationcited by 10
  9. 2025HiPrune: Hierarchical Attention for Efficient Token Pruning in Vision-Language Modelscited by 10
  10. 2022X2-VLM: All-In-One Pre-trained Model For Vision-Language Taskscited by 9
  11. 2025MiCoTA: Bridging the Learnability Gap with Intermediate CoT and Teacher Assistantscited by 9
  12. 2023SmartTrim: Adaptive Tokens and Attention Pruning for Efficient Vision-Language Modelscited by 7
  13. 2025VEAttack: Downstream-agnostic Vision Encoder Attack against Large Vision Language Modelscited by 7
  14. 2023DLIP: Distilling Language-Image Pre-trainingcited by 5
  15. 2023Parameter and Computation Efficient Transfer Learning for Vision-Language Pre-trained Modelscited by 5
  16. 2024Leave No Knowledge Behind During Knowledge Distillation: Towards Practical and Effective Knowledge Distillation for Code-Switching ASR Using Realistic Datacited by 5
  17. 2025LEGO Co-builder: Exploring Fine-Grained Vision-Language Modeling for Multimodal LEGO Assembly Assistantscited by 5
  18. 2025VLM-Pruner: Buffering for Spatial Sparsity in an Efficient VLM Centrifugal Token Pruning Paradigmcited by 5
  19. 2023Modular Transformers: Compressing Transformers into Modularized Layers for Flexible Efficient Inferencecited by 4
  20. 2025TinyAlign: Boosting Lightweight Vision-Language Models by Mitigating Modal Alignment Bottleneckscited by 4
  21. 2025Towards Efficient Agents: A Co-Design of Inference Architecture and Systemcited by 4
  22. 2024Zoom-shot: Fast and Efficient Unsupervised Zero-Shot Transfer of CLIP to Vision Encoders with Multimodal Losscited by 3
  23. 2025EffiVLM-BENCH: A Comprehensive Benchmark for Evaluating Training-Free Acceleration in Large Vision-Language Modelscited by 3
  24. 2025HKD4VLM: A Progressive Hybrid Knowledge Distillation Framework for Robust Multimodal Hallucination and Factuality Detection in VLMscited by 3
  25. 2025DeepInsert: Early Layer Bypass for Efficient and Performant Multimodal Understandingcited by 2
  26. 2025Bridging Ears and Eyes: Analyzing Audio and Visual Large Language Models to Humans in Visible Sound Recognition and Reducing Their Sensory Gap via Cross-Modal Distillationcited by 2
  27. 2025Online In-Context Distillation for Low-Resource Vision Language Modelscited by 2
  28. 2025HyperVL: An Efficient and Dynamic Multimodal Large Language Model for Edge Devicescited by 2
  29. 2026MI-Pruner: Crossmodal Mutual Information-guided Token Pruner for Efficient MLLMscited by 2
  30. 2026Pro2Assist: Continuous Step-aware Proactive Assistance with Multi-modal Egocentric Perception for Long-horizon Procedural Taskscited by 2
  31. 2023MCAD: Multi-teacher Cross-modal Alignment Distillation for efficient image-text retrievalcited by 1
  32. 2025Harnessing Input-Adaptive Inference for Efficient VLNcited by 1
  33. 2025FastVLM: Self-Speculative Decoding for Fast Vision-Language Model Inferencecited by 1
  34. 2026Prune Once: Retraining-Free Task-Agnostic Pruning for Vision-Language Modelscited by 1
  35. 2023Efficient Image-Text Retrieval via Keyword-Guided Pre-Screeningcited by 0
  36. 2023E-ViLM: Efficient Video-Language Model via Masked Video Modeling with Semantic Vector-Quantized Tokenizercited by 0
  37. 2025Hierarchy-Aware Fine-Tuning of Vision-Language Modelscited by 0
  38. 2026CLIP-Map: Structured Matrix Mapping for Parameter-Efficient CLIP Compressioncited by 0

50 most cited of 56 All 56 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Short-LVLM: Compressing and Accelerating Large Vision-Language Models by Pruning Redundant Layers
  2. 2026SlimVLM: Sensitivity-aware Dynamic Structured Pruning with Adaptive Visual Token Selection for Efficient Vision-Language Models
  3. 2024Rethinking Pruning for Vision-Language Models: Strategies for Effective Sparsity and Performance Restoration
  4. 2026Structural Pruning of Large Vision Language Models: A Comprehensive Study on Pruning Dynamics, Recovery, and Data Efficiency
  5. 2026SwiftVLM: Efficient Vision-Language Model Inference via Cross-Layer Token Bypass
  6. 2023PuMer: Pruning and Merging Tokens for Efficient Vision Language Models
  7. 2026Empirical Recipes for Efficient and Compact Vision-Language Models
  8. 2025A Survey on Efficient Vision-Language Models