mlsanityData to 14 Oct 2026

FoveaTree: Information-Density Trees for Query-Guided Visual Token Pruning in Video Large Language Models

arXiv 2610.16744 ↗First posted 12 Oct 2026cs.CVcited by 0

On the map: Efficient LLM inference and serving → Visual token pruning in multimodal models

Abstract (arXiv)

Video large language models (Video LLMs) encode each frame into hundreds of visual tokens, making video inference expensive. Existing methods reduce this cost through similarity-based merging or attention-based selection but face two challenges: Hierarchy-blind Pruning, which overlooks regional compressibility and can discard critical evidence, and Attention Drift, where positional effects and attention sinks shift text-to-vision attention away from task relevance. We propose FoveaTree, a two-stage, training-free visual-token pruning framework. First, Merge-Gain Density Estimation measures the feature distortion caused by regional merging and constructs a quadtree-derived density map to guide token dropping, merging, and retention. Second, Attention Tree Pruning uses the surviving units and inherited tree topology to refine text-to-vision attention in intermediate LLM layers, preserving relevant regional evidence while removing low-relevance content. We evaluate FoveaTree on InternVL3.5-8B and Qwen3-VL-8B across five video question-answering benchmarks and one captioning benchmark at 50%, 30%, and 10% visual-token retention. FoveaTree consistently achieves overall state-of-the-art performance across the evaluated settings, offering a strong accuracy-efficiency trade-off.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Visual token pruning in multimodal models, one of 12 directions in Efficient LLM inference and serving. Builds on 80 of its 90 references that are in the map. First posted on arXiv 12 Oct 2026.

Builds on

80 of 90 references.

Closest ancestors · 19

Same direction, newest first

  1. 2026Audio-Visual Exchange-Aware Token Pruning for Efficient Audio-Visual Captioning9 Jun
  2. 2026FlashVID: Efficient Video Large Language Models via Training-free Tree-based Spatiotemporal Token Merging8 Feb
  3. 2025D2Pruner: Debiased Importance and Structural Diversity for MLLM Token Pruning22 Dec
  4. 2025Don't Just Chase "Highlighted Tokens" in MLLMs: Revisiting Visual Holistic Context Retention3 Oct
  5. 2025Attention Debiasing for Token Pruning in Vision Language Models25 Aug
  6. 2025Multi-Granular Spatio-Temporal Token Merging for Training-Free Acceleration of Video LLMs10 Jul
  7. 2025Stop Looking for Important Tokens in Multimodal Language Models: Duplication Matters More17 Feb
  8. 2024FrameFusion: Combining Similarity and Importance for Video Token Reduction on Large Vision Language Models30 Dec
  9. 2024Feather the Throttle: Revisiting Visual Token Pruning for Vision-Language Model Acceleration17 Dec
  10. 2024Beyond Text-Visual Attention: Exploiting Visual Cues for Effective Token Pruning in VLMs2 Dec
  11. 2024ATP-LLaVA: Adaptive Token Pruning for Large Vision Language Models30 Nov
  12. 2024FocusLLaVA: A Coarse-to-Fine Approach for Efficient and Effective Visual Token Compression21 Nov
7 more
  1. 2024VL-Cache: Sparsity and Modality-Aware KV Cache Compression for Vision-Language Model Inference Acceleration29 Oct
  2. 2024ZipVL: Efficient Large Vision-Language Models with Dynamic Token Sparsification11 Oct
  3. 2024HiRED: Attention-Guided Token Dropping for Efficient Inference of High-Resolution Vision-Language Models20 Aug
  4. 2024DeCo: Decoupling Token Compression from Semantic Abstraction in Multimodal Large Language Models31 May
  5. 2024LLaVA-PruMerge: Adaptive Token Reduction for Efficient Large Multimodal Models22 Mar
  6. 2024An Image is Worth 1/2 Tokens After Layer 2: Plug-and-Play Inference Acceleration for Large Vision-Language Models11 Mar
  7. 2024MADTP: Multimodal Alignment-Guided Dynamic Token Pruning for Accelerating Vision-Language Transformer5 Mar

Other references · 61

  1. 2017Attention Is All You Needcited by 60,361
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  4. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,139
  5. 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,974
  6. 2023Visual Instruction Tuningcited by 7,869
  7. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,403
  8. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,968
  9. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
  10. 2025Qwen2.5-VL Technical Reportcited by 5,587
  11. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,703
  12. 2023Improved Baselines with Visual Instruction Tuningcited by 4,418
38 more
  1. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,386
  2. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,322
  3. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,234
  4. 2023Lost in the Middle: How Language Models Use Long Contextscited by 3,455
  5. 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,109
  6. 2025Qwen3-VL Technical Reportcited by 2,793
  7. 2024LLaVA-OneVision: Easy Visual Task Transfercited by 2,719
  8. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Taskscited by 2,565
  9. 2023Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyondcited by 2,224
  10. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,186
  11. 2023MMBench: Is Your Multi-modal Model an All-around Player?cited by 2,141
  12. 2025InternVL3.5: Advancing Open-Source Multimodal Models in Versatility, Reasoning, and Efficiencycited by 1,510
  13. 2024Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysiscited by 1,413
  14. 2023Video-LLaVA: Learning United Visual Representation by Alignment Before Projectioncited by 1,402
  15. 2023MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilitiescited by 1,239
  16. 2023MVBench: A Comprehensive Multi-modal Video Understanding Benchmarkcited by 1,109
  17. 2023Video-ChatGPT: Towards Detailed Video Understanding via Large Vision and Language Modelscited by 1,062
  18. 2024How Far Are We to GPT-4V? Closing the Gap to Commercial Multimodal Models with Open-Source Suitescited by 1,050
  19. 2022Token Merging: Your ViT But Fastercited by 1,021
  20. 2023ShareGPT4V: Improving Large Multi-Modal Models with Better Captionscited by 1,005
  21. 2021DynamicViT: Efficient Vision Transformers with Dynamic Token Sparsificationcited by 864
  22. 2021NExT-QA:Next Phase of Question-Answering to Explaining Temporal Actionscited by 744
  23. 2024Eyes Wide Shut? Exploring the Visual Shortcomings of Multimodal LLMscited by 735
  24. 2023LLaMA-VID: An Image is Worth 2 Tokens in Large Language Modelscited by 546
  25. 2023TimeChat: A Time-sensitive Multimodal Large Language Model for Long Video Understandingcited by 406
  26. 2024SparseVLM: Visual Token Sparsification for Efficient Vision-Language Model Inferencecited by 391
  27. 2023Chat-UniVi: Unified Visual Representation Empowers Large Language Models with Image and Video Understandingcited by 371
  28. 2024Mini-Gemini: Mining the Potential of Multi-modality Vision Language Modelscited by 365
  29. 2022Not All Patches are What You Need: Expediting Vision Transformers via Token Reorganizationscited by 363
  30. 2024InternVideo2: Scaling Foundation Models for Multimodal Video Understandingcited by 317
  31. 2024VisionZip: Longer is Better but Not Necessary in Vision Language Modelscited by 313
  32. 2024PyramidDrop: Accelerating Your Large Vision-Language Models via Pyramid Visual Redundancy Reductioncited by 294
  33. 2024LLaVA-UHD: an LMM Perceiving Any Aspect Ratio and High-Resolution Imagescited by 195
  34. 2021Adaptive Token Sampling For Efficient Vision Transformerscited by 183
  35. 2023VALOR: Vision-Audio-Language Omni-Perception Pretraining Model and Datasetcited by 140
  36. 2024AuroraCap: Efficient, Performant Video Detailed Captioning and a New Benchmarkcited by 140
  37. 2024Towards Interpreting Visual Information Processing in Vision-Language Modelscited by 122
  38. 2021Token Pooling in Vision Transformerscited by 69

50 most cited of 61 All 61 (JSON)

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2024PruneVid: Visual Token Pruning for Efficient Video Large Language Models
  2. 2026IWP: Token Pruning as Implicit Weight Pruning in Large Vision Language Models
  3. 2025ShaRP: SHAllow-LayeR Pruning for Efficient Video Large Language Models
  4. 2026TReVS: Integrating Textual Relevance and Visual Saliency for Efficient Vision-Language Model Token Pruning
  5. 2025AdaTP: Attention-Debiased Token Pruning for Video Large Language Models
  6. 2026Sink-Token-Aware Pruning for Fine-Grained Video Understanding in Efficient Video LLMs
  7. 2025ToDRE: Effective Visual Token Pruning via Token Diversity and Task Relevance
  8. 2026LearnPruner: Rethinking Attention-based Token Pruning in Vision Language Models