mlsanityData to 14 Oct 2026

LiteVLM: A Low-Latency Vision-Language Model Inference Pipeline for Resource-Constrained Environments

arXiv 2506.07416 ↗First posted 9 Jun 2025cs.LG cs.AIcited by 4

On the map: Multimodal LLMs / vision-language models → Visual token pruning and efficiency

Abstract (arXiv)

This paper introduces an efficient Vision-Language Model (VLM) pipeline specifically optimized for deployment on embedded devices, such as those used in robotics and autonomous driving. The pipeline significantly reduces the computational overhead by jointly leveraging patch selection to filter irrelevant camera views, a token selection module to reduce input sequence length for the LLM, and speculative decoding to accelerate token generation. Evaluation on the NVIDIA DRIVE Thor platform for automonous driving application, our pipeline achieves 2.5× end-to-end latency reduction without compromising task accuracy. The speed-up further increases to 3.2× when applying FP8 post-training quantization. These results demonstrate our pipeline as a viable solution for enabling real-time VLM deployment in resource-constrained environments.

Key facts. Cited by 4 papers in the map (data through 14 Oct 2026). Placed in Visual token pruning and efficiency, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 27 of its 36 references that are in the map. First posted on arXiv 9 Jun 2025.

Builds on

27 of 36 references.

Closest ancestors · 5

Same direction, newest first

  1. 2025TokenFLEX: Unified VLM Training for Flexible Visual Tokens Inference4 Apr
  2. 2024AVG-LLaVA: An Efficient Large Multimodal Model with Adaptive Visual Granularity20 Sep
  3. 2024TokenPacker: Efficient Visual Projector for Multimodal LLM2 Jul
  4. 2024LLaVA-PruMerge: Adaptive Token Reduction for Efficient Large Multimodal Models22 Mar
  5. 2024An Image is Worth 1/2 Tokens After Layer 2: Plug-and-Play Inference Acceleration for Large Vision-Language Models11 Mar

Other references · 22

  1. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  2. 2023Visual Instruction Tuningcited by 7,869
  3. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
  4. 2019nuScenes: A multimodal dataset for autonomous drivingcited by 4,523
  5. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,234
  6. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,711
  7. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Taskscited by 2,565
  8. 2023Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyondcited by 2,224
  9. 2014CIDEr: Consensus-based Image Description Evaluationcited by 2,046
  10. 2022Fast Inference from Transformers via Speculative Decodingcited by 1,653
  11. 2024Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scalingcited by 1,599
  12. 2024Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Headscited by 843
10 more
  1. 2023The Dawn of LMMs: Preliminary Explorations with GPT-4V(ision)cited by 681
  2. 2024EAGLE: Speculative Sampling Requires Rethinking Feature Uncertaintycited by 578
  3. 2023DriveLM: Driving with Graph Visual Question Answeringcited by 536
  4. 2024DriveVLM: The Convergence of Autonomous Driving and Large Vision-Language Modelscited by 490
  5. 2023DriveGPT4: Interpretable End-to-end Autonomous Driving via Large Language Modelcited by 461
  6. 2024EAGLE-2: Faster Inference of Language Models with Dynamic Draft Treescited by 370
  7. 2024Eagle: Exploring The Design Space for Multimodal LLMs with Mixture of Encoderscited by 139
  8. 2023Dolphins: Multimodal Language Model for Drivingcited by 124
  9. 2024Mini-InternVL: A Flexible-Transfer Pocket Multimodal Model with 5% Parameters and 90% Performancecited by 94
  10. 2024ToSA: Token Selective Attention for Efficient Vision Transformerscited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 2; 2026: 2.

4 papers.

  1. 2025Efficient Onboard Vision-Language Inference in UAV-Enabled Low-Altitude Economy Networks via LLM-Enhanced Optimizationcited by 2
  2. 2026HomeSafe-Bench: Evaluating Vision-Language Models on Unsafe Action Detection for Embodied Agents in Household Scenarioscited by 1
  3. 2026Adversarial Prompts for Acceptance Collapse in Speculative Decodingcited by 1
  4. 2025Nav-EE: Navigation-Guided Early Exiting for Efficient Vision-Language Models in Autonomous Drivingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Empirical Recipes for Efficient and Compact Vision-Language Models
  2. 2026Llama-Mobile: Efficient 2.7-Bit Quantization of VLMs
  3. 2026LQA: A Lightweight Quantized-Adaptive Framework for Vision-Language Models on the Edge
  4. 2026CIVIC: End-to-End Sequence Compactness for Efficient Vision-Language Models
  5. 2026Efficient Block-Layer Parallel Inference for Vision-Language-Action on Hybrid Architectures
  6. 2023MobileVLM : A Fast, Strong and Open Vision Language Assistant for Mobile Devices
  7. 2026vla.cpp: A Unified Inference Runtime for Vision-Language-Action Models
  8. 2025A Survey on Efficient Vision-Language Models