mlsanityData to 14 Oct 2026

Aligned Vector Quantization for Edge-Cloud Collabrative Vision-Language Models

arXiv 2411.05961 ↗First posted 8 Nov 2024cs.CV cs.AIcited by 0

On the map: Multimodal LLMs / vision-language models → Visual token pruning and efficiency

Abstract (arXiv)

Vision Language Models (VLMs) are central to Visual Question Answering (VQA) systems and are typically deployed in the cloud due to their high computational demands. However, this cloud-only approach underutilizes edge computational resources and requires significant bandwidth for transmitting raw images. In this paper, we introduce an edge-cloud collaborative VQA system, called LLaVA-AlignedVQ, which features a novel Aligned Vector Quantization algorithm (AlignedVQ) that efficiently compress intermediate features without compromising accuracy to support partitioned execution. Our experiments demonstrate that LLaVA-AlignedVQ achieves approximately 1365x compression rate of intermediate features, reducing data transmission overhead by 96.8% compared to transmitting JPEG90-compressed images to the cloud. LLaVA-AlignedVQ achieves an inference speedup of 2-15x while maintaining high accuracy, remaining within -2.23% to +1.6% of the original model's accuracy performance across eight VQA datasets, compared to the cloud-only solution.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Visual token pruning and efficiency, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 26 of its 36 references that are in the map. First posted on arXiv 8 Nov 2024.

Builds on

26 of 36 references.

Closest ancestors · 1

Same direction, newest first

  1. 2024MoPE-CLIP: Structured Pruning for Efficient Vision-Language Models with Module-wise Pruning Error Metric12 Mar

Other references · 25

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2023GPT-4 Technical Reportcited by 18,396
  3. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  4. 2023Visual Instruction Tuningcited by 7,869
  5. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
  6. 2017Neural Discrete Representation Learningcited by 4,712
  7. 2023Improved Baselines with Visual Instruction Tuningcited by 4,418
  8. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,386
  9. 2020Taming Transformers for High-Resolution Image Synthesiscited by 3,253
  10. 2022Scaling Instruction-Finetuned Language Modelscited by 3,024
  11. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,516
  12. 2023MMBench: Is Your Multi-modal Model an All-around Player?cited by 2,141
13 more
  1. 2023Evaluating Object Hallucination in Large Vision-Language Modelscited by 1,729
  2. 2019Towards VQA Models That Can Readcited by 1,718
  3. 2023MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilitiescited by 1,239
  4. 2018VizWiz Grand Challenge: Answering Visual Questions from Blind Peoplecited by 858
  5. 2021SoundStream: An End-to-End Neural Audio Codeccited by 771
  6. 2017Distributed Deep Neural Networks over the Cloud, the Edge and End Devicescited by 120
  7. 2023HiFi-Codec: Group-residual Vector quantization for High Fidelity Audio Codeccited by 97
  8. 2019And the Bit Goes Down: Revisiting the Quantization of Neural Networkscited by 78
  9. 2024VPTQ: Extreme Low-bit Vector Post-Training Quantization for Large Language Modelscited by 57
  10. 2018Deep feature compression for collaborative object detectioncited by 48
  11. 2021Lightweight compression of neural network feature tensors for collaborative intelligencecited by 17
  12. 2024VQ4DiT: Efficient Post-Training Vector Quantization for Diffusion Transformerscited by 12
  13. 2021Packet-Loss-Tolerant Split Inference for Delay-Sensitive Deep Learning in Lossy Wireless Networkscited by 7

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025When Big Models Train Small Ones: Label-Free Model Parity Alignment for Efficient Visual Question Answering using Small VLMs
  2. 2026Llama-Mobile: Efficient 2.7-Bit Quantization of VLMs
  3. 2025QG-VTC: Question-Guided Visual Token Compression in MLLMs for Efficient VQA
  4. 2025Bi-VLM: Pushing Ultra-Low Precision Post-Training Quantization Boundaries in Vision-Language Models
  5. 2026Evaluating the Impact of Post-Training Quantization on Reliable VQA with Multimodal LLMs
  6. 2025QSVD: Efficient Low-rank Approximation for Unified Query-Key-Value Weight Compression in Low-Precision Vision-Language Models
  7. 2026INAR-VL: Input-Aware Routing for Edge-Cloud Vision-Language Inference
  8. 2022Compressing And Debiasing Vision-Language Pre-Trained Models for Visual Question Answering