VLMaxxing through FrameMogging Training-Free Anti-Recomputation for Video Vision-Language Models
On the map: Multimodal LLMs / vision-language models → Visual token pruning and efficiency
Abstract (arXiv)
Video vision-language models (VLMs) keep paying for visual state the stream already told us was stable. The factory wall did not move, but most VLM pipelines still hand the model dense RGB frames or a fresh prefix again. We study that waste as training-free anti-recomputation: reuse state when validation says it survives, and buy fresh evidence when the scene, query, or cache topology requires it. The largest measured win is after ingest. On frozen Qwen2.5-VL-7B-Instruct-4bit, adaptive same-video follow-up reuse preserves paired choices and correctness on a 93-query VideoMME breadth setting while reducing follow-up latency by 14.90-35.92x. The first query is still cold; the win starts when later questions reuse the same video state. Stress tests bound the result: repeated-question schedules hold through 50 turns, while dense-answer-anchored prompt variation separates conservative fixed K=1 repair from faster aggressive policies that drift. Fresh-video pruning is smaller but real. C-VISION skips timed vision-tower work before the first answer is generated. On Gemma 4-E4B-4bit, the clean 32f short cell reaches 1.316x first-query speedup with no paired drift or parse failures on 20 items; Qwen shows the fidelity/speed boundary. Stage-share ceiling (C-CEILING) is the accounting guardrail: a component speedup becomes an end-to-end speedup only in proportion to the wall-clock share it accelerates, so C-VISION and after-ingest follow-up reuse do not multiply. Candidate C-STREAM remains a native-rate target, not a headline result here. The broader direction is VLM-native media that expose change, motion, uncertainty, object state, sensor time, and active tiles directly, so models do not have to rediscover the world from dense RGB every frame.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Visual token pruning and efficiency, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 30 of its 38 references that are in the map. First posted on arXiv 5 May 2026.
Builds on
30 of 38 references.
Closest ancestors · 9
Same direction, newest first
- 2026CodecSight: Leveraging Video Codec Signals for Efficient Streaming VLM Inference7 Apr
- 2026EvoPrune: Early-Stage Visual Token Pruning for Efficient MLLMs4 Mar
- 2025SparseVILA: Decoupling Visual Sparsity for Efficient VLM Inference20 Oct
- 2025VScan: Rethinking Visual Token Reduction for Efficient Large Vision-Language Models28 May
- 2024FrameFusion: Combining Similarity and Importance for Video Token Reduction on Large Vision Language Models30 Dec
- 2024VisionZip: Longer is Better but Not Necessary in Vision Language Models5 Dec
- 2024SparseVLM: Visual Token Sparsification for Efficient Vision-Language Model Inference6 Oct
- 2024LLaVA-PruMerge: Adaptive Token Reduction for Efficient Large Multimodal Models22 Mar
- 2024An Image is Worth 1/2 Tokens After Layer 2: Plug-and-Play Inference Acceleration for Large Vision-Language Models11 Mar
Other references · 21
- 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
- 2023Efficient Streaming Language Models with Attention Sinkscited by 2,186
- 2024Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysiscited by 1,413
- 2023SGLang: Efficient Execution of Structured Language Model Programscited by 1,141
- 2023MVBench: A Comprehensive Multi-modal Video Understanding Benchmarkcited by 1,109
- 2022Token Merging: Your ViT But Fastercited by 1,021
- 2024CacheBlend: Fast Large Language Model Serving for RAG with Cached Knowledge Fusioncited by 223
- 2025Streaming Video Question-Answering with In-context Video KV-Cache Retrievalcited by 117
- 2017Compressed Video Action Recognitioncited by 106
- 2025StreamingVLM: Real-Time Understanding for Infinite Video Streamscited by 95
- 2024TOMATO: Assessing Visual Temporal Reasoning Capabilities in Multimodal Foundation Modelscited by 73
- 2025FastVID: Dynamic Density Pruning for Fast Video Large Language Modelscited by 52
9 more
- 2026HERMES: KV Cache as Hierarchical Memory for Efficient Streaming Video Understandingcited by 41
- 2026A Simple Baseline for Streaming Video Understandingcited by 33
- 2025Multi-Granular Spatio-Temporal Token Merging for Training-Free Acceleration of Video LLMscited by 31
- 2026FlashVID: Efficient Video Large Language Models via Training-free Tree-based Spatiotemporal Token Mergingcited by 18
- 2023Eventful Transformers: Leveraging Temporal Redundancy in Vision Transformerscited by 13
- 2026CoPE-VideoLM: Leveraging Codec Primitives For Efficient Video Language Modelingcited by 11
- 2025QuickVideo: Real-Time Long Video Understanding with System Algorithm Co-Designcited by 8
- 2025VLCache: Computing 2% Vision Tokens and Reusing 98% for Vision-Language Inferencecited by 7
- 2025D'ej`a Vu: Efficient Video-Language Query Engine with Learning-based Inter-Frame Computation Reusecited by 5
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2025Frame Sampling Strategies Matter: A Benchmark for small vision language models
- 2025Efficient Video Sampling: Pruning Temporally Redundant Tokens for Faster VLM Inference
- 2026CONDUIT: A Unified Residual-Stream Restoration Framework for KV Cache Reuse in Vision-Language Models
- 2026VETO: Video Efficient Token Optimization for Vision Language Models
- 2025AdaptVision: Efficient Vision-Language Models via Adaptive Visual Acquisition
- 2026Reroute, Don't Remove: Recoverable Visual Token Routing for Vision-Language Models
- 2026Frames2LoRA: Parametric Video Internalization for Vision-Language Models
- 2026ResPrune: Text-Conditioned Subspace Reconstruction for Visual Token Pruning in Large Vision-Language Models