mlsanityData to 7 Oct 2026

Self-Adapting Large Visual-Language Models to Edge Devices across Visual Modalities

arXiv 2403.04908 ↗First posted 7 Mar 2024cs.CVcited by 7

On the map: Multimodal LLMs / vision-language models → Visual token pruning and efficiency

Abstract (arXiv)

Recent advancements in Vision-Language (VL) models have sparked interest in their deployment on edge devices, yet challenges in handling diverse visual modalities, manual annotation, and computational constraints remain. We introduce EdgeVL, a novel framework that bridges this gap by seamlessly integrating dual-modality knowledge distillation and quantization-aware contrastive learning. This approach enables the adaptation of large VL models, like CLIP, for efficient use with both RGB and non-RGB images on resource-limited devices without the need for manual annotations. EdgeVL not only transfers visual language alignment capabilities to compact models but also maintains feature quality post-quantization, significantly enhancing open-vocabulary classification performance across various visual modalities. Our work represents the first systematic effort to adapt large VL models for edge deployment, showcasing up to 15.4% accuracy improvements on multiple datasets and up to 93-fold reduction in model size.

Key facts. Cited by 7 papers in the map (data through 7 Oct 2026). Placed in Visual token pruning and efficiency, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 42 of its 53 references that are in the map. First posted on arXiv 7 Mar 2024.

Builds on

42 of 53 references.

References · 42

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  3. 2023GPT-4 Technical Reportcited by 18,132
  4. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  5. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  6. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  7. 2020Training data-efficient image transformers & distillation through attentioncited by 4,560
  8. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
  9. 2017ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenescited by 3,088
  10. 2015FaceNet: A Unified Embedding for Face Recognition and Clusteringcited by 2,833
  11. 2014FitNets: Hints for Thin Deep Netscited by 1,619
  12. 2017Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inferencecited by 1,548
30 more
  1. 2017EuroSAT: A Novel Dataset and Deep Learning Benchmark for Land Use and Land Cover Classificationcited by 1,488
  2. 2022SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Modelscited by 1,338
  3. 2021Open-vocabulary Object Detection via Vision and Language Knowledge Distillationcited by 903
  4. 2022OpenScene: 3D Scene Understanding with Open Vocabulariescited by 556
  5. 2018Quantizing deep convolutional networks for efficient inference: A whitepapercited by 507
  6. 2021Scaling Open-Vocabulary Image Segmentation with Image-Level Labelscited by 471
  7. 2018PACT: Parameterized Clipping Activation for Quantized Neural Networkscited by 456
  8. 2020Open-Vocabulary Object Detection Using Captionscited by 427
  9. 2019Data-Free Quantization Through Weight Equalization and Bias Correctioncited by 328
  10. 2022Learning to Prompt for Open-Vocabulary Object Detection with Vision-Language Modelcited by 317
  11. 2020ZeroQ: A Novel Zero Shot Quantization Frameworkcited by 265
  12. 2022ULIP: Learning a Unified Representation of Language, Images, and Point Clouds for 3D Understandingcited by 262
  13. 2020Integer Quantization for Deep Learning Inference: Principles and Empirical Evaluationcited by 183
  14. 2023CLIP2Scene: Towards Label-efficient 3D Scene Understanding by CLIPcited by 178
  15. 2022PLA: Language-Driven Open-Vocabulary 3D Scene Understandingcited by 176
  16. 2021General Instance Distillation for Object Detectioncited by 113
  17. 2022Q-ViT: Accurate and Fully Quantized Low-bit Vision Transformercited by 106
  18. 2023FreeSeg: Unified, Universal and Open-Vocabulary Image Segmentationcited by 94
  19. 2021Open-Vocabulary Instance Segmentation via Robust Cross-Modal Pseudo-Labelingcited by 84
  20. 2021Network Quantization with Element-wise Gradient Scalingcited by 82
  21. 2022PD-Quant: Post-Training Quantization based on Prediction Difference Metriccited by 73
  22. 2022Revisiting Random Channel Pruning for Neural Network Compressioncited by 50
  23. 2022Cross-Modality Knowledge Distillation Network for Monocular 3D Object Detectioncited by 48
  24. 2019Cross-modal knowledge distillation for action recognitioncited by 44
  25. 2022Open-Vocabulary Multi-Label Classification via Multi-Modal Knowledge Transfercited by 33
  26. 2022MixSKD: Self-Knowledge Distillation from Mixup for Image Recognitioncited by 30
  27. 2023Simple Image-level Classification Improves Open-vocabulary Object Detectioncited by 15
  28. 2018Cross-Modal Distillation for RGB-Depth Person Re-Identificationcited by 13
  29. 2016Adapting Models to Signal Degradation using Distillationcited by 11
  30. 2023Efficient Multi-Task Scene Analysis with RGB-D Transformerscited by 7

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 2; 2025: 3; 2026: 2.

7 papers.

  1. 2025Vision-Language Models for Edge Networks: A Comprehensive Surveycited by 29
  2. 2024Large Model for Small Data: Foundation Model for Cross-Modal RF Human Activity Recognitioncited by 6
  3. 2024FlexiFly: Interfacing the Physical World with Foundation Models Empowered by Reconfigurable Drone Systemscited by 3
  4. 2025Bridging Human Oversight and Black-box Driver Assistance: Vision-Language Models for Predictive Alerting in Lane Keeping Assist Systemscited by 2
  5. 2025Enabling Near-realtime Remote Sensing via Satellite-Ground Collaboration of Large Vision-Language Modelscited by 1
  6. 2026LLMs Explain't: A Post-Mortem on Semantic Interpretability in Transformer Modelscited by 0
  7. 2026Efficient Quantization-Aware Distillation with Cross-Modal Alignment for Edge Vision-Language Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025From Pixels to Words -- Towards Native Vision-Language Primitives at Scale
  2. 2026LQA: A Lightweight Quantized-Adaptive Framework for Vision-Language Models on the Edge
  3. 2025A Survey of State of the Art Large Vision Language Models: Alignment, Benchmark, Evaluations and Challenges
  4. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Tasks
  5. 2025Adapting Vision-Language Models Without Labels: A Comprehensive Survey
  6. 2024DeepSeek-VL: Towards Real-World Vision-Language Understanding
  7. 2025Seeing Beyond Words: Self-Supervised Visual Learning for Multimodal Large Language Models
  8. 2026StructXLIP: Enhancing Vision-language Models with Multimodal Structural Cues