mlsanityData to 7 Oct 2026

VETO: Video Efficient Token Optimization for Vision Language Models

arXiv 2610.01785 ↗First posted 1 Oct 2026cs.CV cs.AI cs.CLcited by 0

On the map: Multimodal LLMs / vision-language models → Visual token pruning and efficiency

Abstract (arXiv)

Processing long videos with Vision-Language Models (VLMs) is bottlenecked by the quadratic cost of visual tokens, making long-form inference prohibitively expensive. While single-axis compression methods mitigate this, they hit a hard efficiency floor because they treat spatial and temporal redundancy independently. We present VETO (Video Efficient Token Optimization for Vision-Language Models), a training-optional plug-in that eliminates this bottleneck through dual-axis compression: (i) an intra-frame compressor that merges semantically similar tokens within each frame via optimal-transport inspired matching, and (ii) an inter-frame compressor that identifies and merges temporally redundant frames. The key design insight is hierarchical ordering: by first compressing spatial dimensions, VETO drastically reduces the cost of subsequent global temporal matching, bypassing the efficiency wall of single-axis approaches, with an advantage that grows with modern fully-fused attention infrastructure. Empirically, VETO achieves up to 45% faster inference (e.g., on LLaVA-OneVision-7B) while preserving or improving accuracy. Under extreme token starvation (10% budget), VETO outperforms VFlowOpt (54.9%), VisionZip (52.6%), and FastV (47.9%) with 55.7% accuracy. We demonstrate universal applicability across LLaVA-OneVision, InternVL-2.5, and LongVA, with zero-shot accuracy preserved or improved in all cases.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Visual token pruning and efficiency, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 32 of its 35 references that are in the map. First posted on arXiv 1 Oct 2026.

Builds on

32 of 35 references.

Closest ancestors · 5

Same direction, newest first

  1. 2025VFlowOpt: A Token Pruning Framework for LMMs with Visual Information Flow-Guided Optimization7 Aug
  2. 2024VisionZip: Longer is Better but Not Necessary in Vision Language Models5 Dec
  3. 2024VideoLLM-MoD: Efficient Video-Language Streaming with Mixture-of-Depths Vision Computation29 Aug
  4. 2024LLaVA-PruMerge: Adaptive Token Reduction for Efficient Large Multimodal Models22 Mar
  5. 2024An Image is Worth 1/2 Tokens After Layer 2: Plug-and-Play Inference Acceleration for Large Vision-Language Models11 Mar

Other references · 27

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2023Visual Instruction Tuningcited by 7,792
  3. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  4. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,855
  5. 2024LLaVA-OneVision: Easy Visual Task Transfercited by 2,669
  6. 2023Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyondcited by 2,179
  7. 2024Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scalingcited by 1,573
  8. 2023Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understandingcited by 1,461
  9. 2022VideoMAE: Masked Autoencoders are Data-Efficient Learners for Self-Supervised Video Pre-Trainingcited by 1,305
  10. 2014Two-Stream Convolutional Networks for Action Recognition in Videoscited by 1,067
  11. 2023Video-ChatGPT: Towards Detailed Video Understanding via Large Vision and Language Modelscited by 1,047
  12. 2022Token Merging: Your ViT But Fastercited by 995
15 more
  1. 2016Temporal Segment Networks: Towards Good Practices for Deep Action Recognitioncited by 899
  2. 2018TSM: Temporal Shift Module for Efficient Video Understandingcited by 852
  3. 2023VideoChat: Chat-Centric Video Understandingcited by 851
  4. 2021DynamicViT: Efficient Vision Transformers with Dynamic Token Sparsificationcited by 849
  5. 2023MovieChat: From Dense Token to Sparse Memory for Long Video Understandingcited by 536
  6. 2024Long Context Transfer from Language to Visioncited by 502
  7. 2023TimeChat: A Time-sensitive Multimodal Large Language Model for Long Video Understandingcited by 399
  8. 2024MA-LMM: Memory-Augmented Large Multimodal Model for Long-Term Video Understandingcited by 223
  9. 2024ST-LLM: Large Language Models Are Effective Temporal Learnerscited by 126
  10. 2022Long Movie Clip Classification with State-Space Video Modelscited by 106
  11. 2023Selective Structured State-Spaces for Long-Form Video Understandingcited by 105
  12. 2024LongLLaVA: Scaling Multi-modal LLMs to 1000 Images Efficiently via a Hybrid Architecturecited by 100
  13. 2022Efficient Movie Scene Detection using State-Space Transformerscited by 51
  14. 2022Movies2Scenes: Using Movie Metadata to Learn Scene Representationcited by 11
  15. 2024HERMES: temporal-coHERent long-forM understanding with Episodes and Semanticscited by 11

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026CRAFT: Compression via Recursive Adaptive Fusion of Video Tokens for Vision-Language Models
  2. 2026Efficient Inference for Large Vision-Language Models: Bottlenecks, Techniques, and Prospects
  3. 2026Token-Budget Distillation: Transferring Full-Token Semantics to Compressed Video Vision-Language Models
  4. 2025Variation-aware Vision Token Dropping for Faster Large Vision-Language Models
  5. 2025Hybrid Token Compression for Vision-Language Models
  6. 2026One Token per Highly Selective Frame: Towards Extreme Compression for Long Video Understanding
  7. 2026AsymVLM: Asymmetric Token Pruning for Efficient Vision-Language Model Inference
  8. 2025A Glimpse to Compress: Dynamic Visual Token Pruning for Large Vision-Language Models