# FrameFusion: Combining Similarity and Importance for Video Token Reduction on Large Vision Language Models

arXiv: https://arxiv.org/abs/2501.01986 · HTML: https://mlsanity.com/paper/2501.01986/

## Key facts

- Cited by 54 papers in the map (data through 14 Oct 2026).
- Placed in Visual token pruning and efficiency, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 23 of its 45 references that are in the map.
- First posted on arXiv 30 Dec 2024.

## Placed in

- Multimodal LLMs / vision-language models / [Visual token pruning and efficiency](https://mlsanity.com/mllm/visual-token-pruning-and-efficiency/)
- Efficient LLM inference and serving / [Visual token pruning in multimodal models](https://mlsanity.com/inference/visual-token-pruning-in-multimodal-models/)

## Builds on (most cited first, 23 of 23)

- [Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context](https://mlsanity.com/paper/2403.05530/) (2024)
- [LLaVA-OneVision: Easy Visual Task Transfer](https://mlsanity.com/paper/2408.03326/) (2024)
- [Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysis](https://mlsanity.com/paper/2405.21075/) (2024)
- [MiniCPM-V: A GPT-4V Level MLLM on Your Phone](https://mlsanity.com/paper/2408.01800/) (2024)
- [SnapKV: LLM Knows What You are Looking for Before Generation](https://mlsanity.com/paper/2404.14469/) (2024)
- [LLaVA-Video: Video Instruction Tuning With Synthetic Data](https://mlsanity.com/paper/2410.02713/) (2024)
- [Quest: Query-Aware Sparsity for Efficient Long-Context LLM Inference](https://mlsanity.com/paper/2406.10774/) (2024)
- [PyramidKV: Dynamic KV Cache Compression based on Pyramidal Information Funneling](https://mlsanity.com/paper/2406.02069/) (2024)
- [LLaVA-PruMerge: Adaptive Token Reduction for Efficient Large Multimodal Models](https://mlsanity.com/paper/2403.15388/) (2024)
- [SparseVLM: Visual Token Sparsification for Efficient Vision-Language Model Inference](https://mlsanity.com/paper/2410.04417/) (2024)
- [MInference 1.0: Accelerating Pre-filling for Long-Context LLMs via Dynamic Sparse Attention](https://mlsanity.com/paper/2407.02490/) (2024)
- [NVILA: Efficient Frontier Visual Language Models](https://mlsanity.com/paper/2412.04468/) (2024)
- [InternLM-XComposer2-4KHD: A Pioneering Large Vision-Language Model Handling Resolutions from 336 Pixels to 4K HD](https://mlsanity.com/paper/2404.06512/) (2024)
- [AuroraCap: Efficient, Performant Video Detailed Captioning and a New Benchmark](https://mlsanity.com/paper/2410.03051/) (2024)
- [SlowFast-LLaVA: A Strong Training-Free Baseline for Video Large Language Models](https://mlsanity.com/paper/2407.15841/) (2024)
- [LM-Infinite: Zero-Shot Extreme Length Generalization for Large Language Models](https://mlsanity.com/paper/2308.16137/) (2023)
- [Efficient Multimodal Large Language Models: A Survey](https://mlsanity.com/paper/2405.10739/) (2024)
- [LazyLLM: Dynamic Token Pruning for Efficient Long Context LLM Inference](https://mlsanity.com/paper/2407.14057/) (2024)
- [Mixture of Attention Spans: Optimizing LLM Inference Efficiency with Heterogeneous Sliding-Window Lengths](https://mlsanity.com/paper/2406.14909/) (2024)
- [VL-Cache: Sparsity and Modality-Aware KV Cache Compression for Vision-Language Model Inference Acceleration](https://mlsanity.com/paper/2410.23317/) (2024)
- [AIM: Adaptive Inference of Multi-Modal LLMs via Token Merging and Pruning](https://mlsanity.com/paper/2412.03248/) (2024)
- [Needle In A Video Haystack: A Scalable Synthetic Evaluator for Video MLLMs](https://mlsanity.com/paper/2406.09367/) (2024)
- [A-VL: Adaptive Attention for Large Vision-Language Models](https://mlsanity.com/paper/2409.14846/) (2024)

## Built on by (most cited first, 30 of 54)

- [FastVID: Dynamic Density Pruning for Fast Video Large Language Models](https://mlsanity.com/paper/2503.11187/) (2025)
- [Video-MMLU: A Massive Multi-Discipline Lecture Understanding Benchmark](https://mlsanity.com/paper/2504.14693/) (2025)
- [Global Compression Commander: Plug-and-Play Inference Acceleration for High-Resolution Large Vision-Language Models](https://mlsanity.com/paper/2501.05179/) (2025)
- [Multi-Granular Spatio-Temporal Token Merging for Training-Free Acceleration of Video LLMs](https://mlsanity.com/paper/2507.07990/) (2025)
- [FluxMem: Adaptive Hierarchical Memory for Streaming Video Understanding](https://mlsanity.com/paper/2603.02096/) (2026)
- [LLaVA-Scissor: Token Compression with Semantic Connected Components for Video LLMs](https://mlsanity.com/paper/2506.21862/) (2025)
- [A Survey of Token Compression for Efficient Multimodal Large Language Models](https://mlsanity.com/paper/2507.20198/) (2025)
- [FlexSelect: Flexible Token Selection for Efficient Long Video Understanding](https://mlsanity.com/paper/2506.00993/) (2025)
- [FlashVID: Efficient Video Large Language Models via Training-free Tree-based Spatiotemporal Token Merging](https://mlsanity.com/paper/2602.08024/) (2026)
- [PAROAttention: Pattern-Aware ReOrdering for Efficient Sparse and Quantized Attention in Visual Generation Models](https://mlsanity.com/paper/2506.16054/) (2025)
- [CoPE-VideoLM: Leveraging Codec Primitives For Efficient Video Language Modeling](https://mlsanity.com/paper/2602.13191/) (2026)
- [Causality Matters: How Temporal Information Emerges in Video Language Models](https://mlsanity.com/paper/2508.11576/) (2025)
- [Breaking the Encoder Barrier for Seamless Video-Language Understanding](https://mlsanity.com/paper/2503.18422/) (2025)
- [ResAdapt: Adaptive Resolution for Efficient Multimodal Reasoning](https://mlsanity.com/paper/2603.28610/) (2026)
- [DynTok: Dynamic Compression of Visual Tokens for Efficient and Effective Video Understanding](https://mlsanity.com/paper/2506.03990/) (2025)
- [LLMC+: Benchmarking Vision-Language Model Compression with a Plug-and-play Toolkit](https://mlsanity.com/paper/2508.09981/) (2025)
- [UniComp: Rethinking Video Compression Through Informational Uniqueness](https://mlsanity.com/paper/2512.03575/) (2025)
- [AdaCodec: A Predictive Visual Code for Video MLLMs](https://mlsanity.com/paper/2606.02569/) (2026)
- [METok: Multi-Stage Event-based Token Compression for Efficient Long Video Understanding](https://mlsanity.com/paper/2506.02850/) (2025)
- [SeViCES: Unifying Semantic-Visual Evidence Consensus for Long Video Understanding](https://mlsanity.com/paper/2510.20622/) (2025)
- [Token Reduction via Local and Global Contexts Optimization for Efficient Video Large Language Models](https://mlsanity.com/paper/2603.01400/) (2026)
- [ForestPrune: High-ratio Visual Token Compression for Video Multimodal Large Language Models via Spatial-Temporal Forest Modeling](https://mlsanity.com/paper/2603.22911/) (2026)
- [DocPrune:Efficient Document Question Answering via Background, Question, and Comprehension-aware Token Pruning](https://mlsanity.com/paper/2604.22281/) (2026)
- [PruneHal: Reducing Hallucinations in Multi-modal Large Language Models through Adaptive KV Cache Pruning](https://mlsanity.com/paper/2510.19183/) (2025)
- [Test-Time Temporal Sampling for Efficient MLLM Video Understanding](https://mlsanity.com/paper/2511.17945/) (2025)
- [Focus: A Streaming Concentration Architecture for Efficient Vision-Language Models](https://mlsanity.com/paper/2512.14661/) (2025)
- [ViKey: Enhancing Temporal Understanding in Videos via Visual Prompting](https://mlsanity.com/paper/2603.23186/) (2026)
- [OmniRefine: Alignment-Aware Cooperative Compression for Efficient Omnimodal Large Language Models](https://mlsanity.com/paper/2605.12056/) (2026)
- [Video Parallel Scaling: Aggregating Diverse Frame Subsets for VideoLLMs](https://mlsanity.com/paper/2509.08016/) (2025)
- [HieraVid: Hierarchical Token Pruning for Fast Video Large Language Models](https://mlsanity.com/paper/2604.01881/) (2026)

## Abstract (arXiv)

The increasing demand to process long and high-resolution videos significantly burdens Large Vision-Language Models (LVLMs) due to the enormous number of visual tokens. Existing token reduction methods primarily prune tokens based on importance metrics, such as cumulative attention scores. However, even important tokens may exhibit high redundancy caused by similarity among adjacent video frames and repetitive visual elements. To address this limitation, we propose FrameFusion, a novel token reduction approach integrating similarity-based merging with importance-based pruning. We conduct a thorough study on token similarity characteristics, revealing three key insights: (1) spatially corresponding visual tokens between adjacent frames have higher cosine similarities compared to other token pairs; (2) high token similarities prominently decrease in deeper model layers; and (3) token similarity rankings are highly consistent across different layers. Guided by these observations, FrameFusion computes token similarities exclusively between corresponding visual tokens from adjacent frames, applies token merging at initial successive layers followed by pruning in deeper layers, and adopts a cascaded merging strategy to further enhance efficiency. We evaluate FrameFusion comprehensively across six diverse LVLMs, ranging from 2B to 72B parameters, using five video benchmarks encompassing video retrieval, question-answering, and spatial-temporal understanding tasks. Experiments show that FrameFusion reduces visual tokens by 70%, achieving 1.6-3.6x end-to-end speedups, with an average performance impact of less than 3%. Our code is available at: https://github.com/thu-nics/FrameFusion.