mlsanityData to 7 Oct 2026

Fre-Res: Frequency-Residual Video Token Compression for Efficient Video MLLMs

arXiv 2605.16366 ↗First posted 10 May 2026cs.CV cs.AIcited by 0

On the map: Efficient LLM inference and serving → Visual token pruning in multimodal models

Abstract (arXiv)

Video MLLMs face a persistent tension between spatial fidelity and temporal coverage: preserving fine-grained visual details requires many spatial tokens, while capturing short-lived events requires dense temporal sampling. We propose Fre-Res, a budget-adaptive dual-track video-token compression framework that separates these two forms of evidence. Fre-Res preserves sparse high-fidelity spatial anchors and represents dense temporal evolution through compact residual-frequency tokens. Specifically, it applies temporal 1D-DCT to inter-frame residual trajectories in vision-latent space, where we observe strong low-frequency concentration. To align frequency-domain dynamics with native visual embeddings, Fre-Res introduces a Spatial-Guided Absorber that injects temporal residual information into spatially corresponding anchor tokens. Across fine-grained short-video and long-video reasoning benchmarks, Fre-Res achieves a favorable accuracy--efficiency trade-off, matching or approaching full-token performance while substantially reducing visual-token length. Extensive ablations further show that temporal-frequency residuals preserve causal transition cues, while spatial anchors remain essential for fine-grained object and layout reasoning.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Visual token pruning in multimodal models, one of 12 directions in Efficient LLM inference and serving. Builds on 26 of its 30 references that are in the map. First posted on arXiv 10 May 2026.

Builds on

26 of 30 references.

Closest ancestors · 7

Same direction, newest first

  1. 2026EvoPrune: Early-Stage Visual Token Pruning for Efficient MLLMs4 Mar
  2. 2025Beyond Attention or Similarity: Maximizing Conditional Diversity for Token Pruning in MLLMs12 Jun
  3. 2025HoliTom: Holistic Token Merging for Fast Video Large Language Models27 May
  4. 2025Token Reduction Should Go Beyond Efficiency in Generative Models -- From Vision, Language to Multimodality23 May
  5. 2024FrameFusion: Combining Similarity and Importance for Video Token Reduction on Large Vision Language Models30 Dec
  6. 2024LLaVA-PruMerge: Adaptive Token Reduction for Efficient Large Multimodal Models22 Mar
  7. 2024An Image is Worth 1/2 Tokens After Layer 2: Plug-and-Play Inference Acceleration for Large Vision-Language Models11 Mar

Other references · 19

  1. 2024The Llama 3 Herd of Modelscited by 14,192
  2. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,151
  3. 2025Qwen3-VL Technical Reportcited by 2,534
  4. 2024Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scalingcited by 1,573
  5. 2023MVBench: A Comprehensive Multi-modal Video Understanding Benchmarkcited by 1,086
  6. 2024MiniCPM-V: A GPT-4V Level MLLM on Your Phonecited by 1,061
  7. 2024SnapKV: LLM Knows What You are Looking for Before Generationcited by 848
  8. 2021NExT-QA:Next Phase of Question-Answering to Explaining Temporal Actionscited by 735
  9. 2023EgoSchema: A Diagnostic Benchmark for Very Long-form Video Language Understandingcited by 666
  10. 2024LongVideoBench: A Benchmark for Long-context Interleaved Video-Language Understandingcited by 644
  11. 2024LLaVA-Video: Video Instruction Tuning With Synthetic Datacited by 550
  12. 2024Long Context Transfer from Language to Visioncited by 502
7 more
  1. 2024LVBench: An Extreme Long Video Understanding Benchmarkcited by 402
  2. 2021FNet: Mixing Tokens with Fourier Transformscited by 343
  3. 2024Leave No Context Behind: Efficient Infinite Context Transformers with Infini-attentioncited by 194
  4. 2024MuseTalk: Real-Time High-Fidelity Video Dubbing via Spatio-Temporal Samplingcited by 27
  5. 2024Learning to Merge Tokens via Decoupled Embedding for Efficient Vision Transformerscited by 21
  6. 2025Learning to Inference Adaptively for Multimodal Large Language Modelscited by 12
  7. 2025FreqKV: Key-Value Compression in Frequency Domain for Context Window Extensioncited by 7

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Rethinking Video Token Compression with a Global Codebook: Learning Once, Compressing Everywhere
  2. 2025Fourier Compressor: Frequency-Domain Visual Token Compression for Vision-Language Models
  3. 2025Progressive Growing of Video Tokenizers for Temporally Compact Latent Spaces
  4. 2026Rethinking Visual Token Compression for Video Large Language Models: A Simple Yet Strong Baseline
  5. 2025Dense Video Understanding with Gated Residual Tokenization
  6. 2026TTF: Temporal Token Fusion for Efficient Video-Language Model
  7. 2026DynaTok: Temporally Adaptive and Positional Bias-Aware Token Compression for Video-LLMs
  8. 2026Compression as Adaptation: Implicit Visual Representation with Diffusion Foundation Models