mlsanityData to 14 Oct 2026

Efficient Quantization-Aware Distillation with Cross-Modal Alignment for Edge Vision-Language Models

arXiv 2609.16689 ↗First posted 15 Sep 2026cs.CVcited by 0

On the map: Multimodal LLMs / vision-language models → Visual token pruning and efficiency

Abstract (arXiv)

Large-scale vision-language models (VLM) such as CLIP enable strong open-vocabulary reasoning, yet deploying these capabilities on resource-constrained edge devices remains challenging. EdgeVL addresses this problem by distilling CLIP representations into lightweight multi-modal encoders and applying quantization-aware training (QAT) for efficient Open-Vocabulary Classification (OVC) on edge hardware. However, its two-stage optimization applies different objectives for distillation and QAT, and contrastive learning is performed within the quantized student space, which can result in inconsistent optimization and reduced training efficiency. Moreover, identical supervision across RGB and non-RGB modalities may lead to modality imbalance. We propose a unified framework for quantized semantic distillation tailored to edge deployment. By jointly optimizing distillation and quantization within a unified teacher-anchored framework, our method ensures consistent training under quantization, suppressing hard negatives and enlarging decision margins. Additionally, we design a lightweight cross-attention adapter that enhances non-RGB representations through RGB-guided semantic transfer, narrowing the modality gap. Extensive experiments demonstrate consistent improvements on non-RGB modalities while maintaining deployment efficiency.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Visual token pruning and efficiency, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 30 of its 37 references that are in the map. First posted on arXiv 15 Sep 2026.

Builds on

30 of 37 references.

Closest ancestors · 1

Same direction, newest first

  1. 2024Self-Adapting Large Visual-Language Models to Edge Devices across Visual Modalities7 Mar

Other references · 29

  1. 2017Attention Is All You Needcited by 60,361
  2. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,139
  3. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  4. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,864
  5. 2017ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenescited by 3,117
  6. 2015Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Codingcited by 2,329
  7. 2017Decoupled Weight Decay Regularizationcited by 1,831
  8. 2020Knowledge Distillation: A Surveycited by 1,603
  9. 2017Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inferencecited by 1,593
  10. 2017EuroSAT: A Novel Dataset and Deep Learning Benchmark for Land Use and Land Cover Classificationcited by 1,496
  11. 2020WILDS: A Benchmark of in-the-Wild Distribution Shiftscited by 1,375
  12. 2016Paying More Attention to Attention: Improving the Performance of Convolutional Neural Networks via Attention Transfercited by 1,077
17 more
  1. 2019Relational Knowledge Distillationcited by 889
  2. 2021Domain Generalization: A Surveycited by 764
  3. 2019Learned Step Size Quantizationcited by 628
  4. 2022Open-Vocabulary Semantic Segmentation with Mask-adapted CLIPcited by 565
  5. 2020Measuring Robustness to Natural Distribution Shifts in Image Classificationcited by 454
  6. 2022Q-ViT: Accurate and Fully Quantized Low-bit Vision Transformercited by 108
  7. 2023CLIP-KD: An Empirical Study of CLIP Model Distillationcited by 87
  8. 2022PD-Quant: Post-Training Quantization based on Prediction Difference Metriccited by 73
  9. 2022Cross-Modality Knowledge Distillation Network for Monocular 3D Object Detectioncited by 48
  10. 2019Cross-modal knowledge distillation for action recognitioncited by 45
  11. 2022EfficientViT: Multi-Scale Linear Attention for High-Resolution Dense Predictioncited by 42
  12. 2022MixSKD: Self-Knowledge Distillation from Mixup for Image Recognitioncited by 30
  13. 2024CLIP-CID: Efficient CLIP Distillation via Cluster-Instance Discriminationcited by 21
  14. 2024LaMI-DETR: Open-Vocabulary Detection with Language Model Instructioncited by 19
  15. 2018Cross-Modal Distillation for RGB-Depth Person Re-Identificationcited by 13
  16. 2016Adapting Models to Signal Degradation using Distillationcited by 11
  17. 2024Open-Vocabulary Object Detection via Language Hierarchycited by 10

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026LQA: A Lightweight Quantized-Adaptive Framework for Vision-Language Models on the Edge
  2. 2026Fine-Grained Post-Training Quantization for Large Vision Language Models with Quantization-Aware Integrated Gradients
  3. 2026Gated Relational Alignment via Confidence-based Distillation for Efficient VLMs
  4. 2025Vision-Language Models for Edge Networks: A Comprehensive Survey
  5. 2026Breaking Modality Heterogeneity in Low-Bit Quantization for Large Vision-Language Models
  6. 2024Q-VLM: Post-training Quantization for Large Vision-Language Models
  7. 2026MiX: Micro-Inverted-Scaling for End-to-End Low-Bit Vision-Language Model Acceleration
  8. 2022EfficientVLM: Fast and Accurate Vision-Language Models via Knowledge Distillation and Modal-adaptive Pruning