# OTT-Vid: Optimal Transport Temporal Token Compression for Video Large Language Models

arXiv: https://arxiv.org/abs/2605.11803 · HTML: https://mlsanity.com/paper/2605.11803/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Placed in Visual token pruning in multimodal models, one of 12 directions in Efficient LLM inference and serving.
- Builds on 31 of its 34 references that are in the map.
- First posted on arXiv 12 May 2026.

## Placed in

- Efficient LLM inference and serving / [Visual token pruning in multimodal models](https://mlsanity.com/inference/visual-token-pruning-in-multimodal-models/)

## Builds on (most cited first, 30 of 31)

- [Qwen2.5-VL Technical Report](https://mlsanity.com/paper/2502.13923/) (2025)
- [Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolution](https://mlsanity.com/paper/2409.12191/) (2024)
- [LLaVA-OneVision: Easy Visual Task Transfer](https://mlsanity.com/paper/2408.03326/) (2024)
- [Sinkhorn Distances: Lightspeed Computation of Optimal Transportation Distances](https://mlsanity.com/paper/1306.0895/) (2013)
- [Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysis](https://mlsanity.com/paper/2405.21075/) (2024)
- [MVBench: A Comprehensive Multi-modal Video Understanding Benchmark](https://mlsanity.com/paper/2311.17005/) (2023)
- [LongVideoBench: A Benchmark for Long-context Interleaved Video-Language Understanding](https://mlsanity.com/paper/2407.15754/) (2024)
- [An Image is Worth 1/2 Tokens After Layer 2: Plug-and-Play Inference Acceleration for Large Vision-Language Models](https://mlsanity.com/paper/2403.06764/) (2024)
- [TALL: Temporal Activity Localization via Language Query](https://mlsanity.com/paper/1705.02101/) (2017)
- [LMMs-Eval: Reality Check on the Evaluation of Large Multimodal Models](https://mlsanity.com/paper/2407.12772/) (2024)
- [VisionZip: Longer is Better but Not Necessary in Vision Language Models](https://mlsanity.com/paper/2412.04467/) (2024)
- [MLVU: Benchmarking Multi-task Long Video Understanding](https://mlsanity.com/paper/2406.04264/) (2024)
- [DivPrune: Diversity-based Visual Token Pruning for Large Multimodal Models](https://mlsanity.com/paper/2503.02175/) (2025)
- [Token Fusion: Bridging the Gap between Token Pruning and Token Merging](https://mlsanity.com/paper/2312.01026/) (2023)
- [DyCoke: Dynamic Compression of Tokens for Fast Video Large Language Models](https://mlsanity.com/paper/2411.15024/) (2024)
- [PruneVid: Visual Token Pruning for Efficient Video Large Language Models](https://mlsanity.com/paper/2412.16117/) (2024)
- [HoliTom: Holistic Token Merging for Fast Video Large Language Models](https://mlsanity.com/paper/2505.21334/) (2025)
- [FrameFusion: Combining Similarity and Importance for Video Token Reduction on Large Vision Language Models](https://mlsanity.com/paper/2501.01986/) (2024)
- [PuMer: Pruning and Merging Tokens for Efficient Vision Language Models](https://mlsanity.com/paper/2305.17530/) (2023)
- [FastVID: Dynamic Density Pruning for Fast Video Large Language Models](https://mlsanity.com/paper/2503.11187/) (2025)
- [TimeLens: Rethinking Video Temporal Grounding with Multimodal LLMs](https://mlsanity.com/paper/2512.14698/) (2025)
- [PACT: Pruning and Clustering-Based Token Reduction for Faster Visual Language Models](https://mlsanity.com/paper/2504.08966/) (2025)
- [Multi-Granular Spatio-Temporal Token Merging for Training-Free Acceleration of Video LLMs](https://mlsanity.com/paper/2507.07990/) (2025)
- [SCOPE: Saliency-Coverage Oriented Token Pruning for Efficient Multimodel LLMs](https://mlsanity.com/paper/2510.24214/) (2025)
- [Prune Redundancy, Preserve Essence: Vision Token Compression in VLMs via Synergistic Importance-Diversity](https://mlsanity.com/paper/2603.09480/) (2026)
- [FlashVID: Efficient Video Large Language Models via Training-free Tree-based Spatiotemporal Token Merging](https://mlsanity.com/paper/2602.08024/) (2026)
- [UniComp: Rethinking Video Compression Through Informational Uniqueness](https://mlsanity.com/paper/2512.03575/) (2025)
- [FLoC: Facility Location-Based Efficient Visual Token Compression for Long Video Understanding](https://mlsanity.com/paper/2511.00141/) (2025)
- [Unified Spatiotemporal Token Compression for Video-LLMs at Ultra-Low Retention](https://mlsanity.com/paper/2603.21957/) (2026)
- [Token Reduction via Local and Global Contexts Optimization for Efficient Video Large Language Models](https://mlsanity.com/paper/2603.01400/) (2026)

## Built on by (most cited first, 1 of 1)

- [Why Is Video Still So Expensive? A Survey of Inference-Efficiency Mechanisms in Video and Audiovisual LLMs](https://mlsanity.com/paper/2609.10355/) (2026)

## Abstract (arXiv)

As Video Large Language Models (Video-LLMs) scale to longer and more complex videos, their inference cost grows rapidly due to the large volume of visual tokens accumulated across frames. Training-free token compression has emerged as a practical solution to this bottleneck. However, existing temporal compression methods rely primarily on cross-frame token similarity or segmentation heuristics, overlooking each token's semantic role within its frame and failing to adapt compression strength to the compressibility of each frame pair. In this work, we propose OTT-Vid, a transport-derived allocation framework for temporal token compression. Our approach consists of two stages: spatial pruning identifies representative content within each frame, and optimal transport (OT) is then solved between neighboring frames to estimate temporal compressibility. We formulate this OT with non-uniform token mass, which protects semantically important tokens from aggressive compression, and a locality-aware cost that captures both feature and spatial disparities. The resulting transport plan jointly balances token importance and matching cost, while its total cost defines the transport difficulty of each frame pair, which we use to allocate compression budgets dynamically. Experiments on six benchmarks spanning video question answering and temporal grounding show that OTT-Vid preserves 95.8% of VQA and 73.9% of VTG performance while retaining only 10% of tokens, consistently outperforming existing state-of-the-art training-free compression methods.