mlsanityData to 7 Oct 2026

vid-TLDR: Training Free Token merging for Light-weight Video Transformer

arXiv 2403.13347 ↗First posted 20 Mar 2024cs.CVcited by 41

Not on the map. Why?

Abstract (arXiv)

Video Transformers have become the prevalent solution for various video downstream tasks with superior expressive power and flexibility. However, these video transformers suffer from heavy computational costs induced by the massive number of tokens across the entire video frames, which has been the major barrier to training the model. Further, the patches irrelevant to the main contents, e.g., backgrounds, degrade the generalization performance of models. To tackle these issues, we propose training free token merging for lightweight video Transformer (vid-TLDR) that aims to enhance the efficiency of video Transformers by merging the background tokens without additional training. For vid-TLDR, we introduce a novel approach to capture the salient regions in videos only with the attention map. Further, we introduce the saliency-aware token merging strategy by dropping the background tokens and sharpening the object scores. Our experiments show that vid-TLDR significantly mitigates the computational complexity of video Transformers while achieving competitive performance compared to the base model without vid-TLDR. Code is available at https://github.com/mlvlab/vid-TLDR.

Key facts. Cited by 41 papers in the map (data through 7 Oct 2026). Builds on 62 of its 77 references that are in the map. First posted on arXiv 20 Mar 2024.

Builds on

62 of 77 references.

References · 62

  1. 2017Attention Is All You Needcited by 59,932
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  4. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  5. 2020End-to-End Object Detection with Transformerscited by 7,384
  6. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,339
  7. 2020Training data-efficient image transformers & distillation through attentioncited by 4,560
  8. 2014Neural Machine Translation by Jointly Learning to Align and Translatecited by 4,530
  9. 2020Deformable DETR: Deformable Transformers for End-to-End Object Detectioncited by 3,410
  10. 2020Longformer: The Long-Document Transformercited by 3,362
  11. 2021SegFormer: Simple and Efficient Design for Semantic Segmentation with Transformerscited by 2,923
  12. 2020Transformers are RNNs: Fast Autoregressive Transformers with Linear Attentioncited by 1,979
38 more
  1. 2020Linformer: Self-Attention with Linear Complexitycited by 1,462
  2. 2022VideoMAE: Masked Autoencoders are Data-Efficient Learners for Self-Supervised Video Pre-Trainingcited by 1,305
  3. 2021Frozen in Time: A Joint Video and Image Encoder for End-to-End Retrievalcited by 1,175
  4. 2017The "something something" video database for learning and evaluating visual common sensecited by 1,092
  5. 2022Token Merging: Your ViT But Fastercited by 995
  6. 2021DynamicViT: Efficient Vision Transformers with Dynamic Token Sparsificationcited by 849
  7. 2020Quantifying Attention Flow in Transformerscited by 759
  8. 2017Localizing Moments in Video with Natural Languagecited by 660
  9. 2021CLIP4Clip: An Empirical Study of CLIP for End to End Video Clip Retrievalcited by 645
  10. 2021Twins: Revisiting the Design of Spatial Attention in Vision Transformerscited by 565
  11. 2023VideoMAE V2: Scaling Video Masked Autoencoders with Dual Maskingcited by 516
  12. 2022DN-DETR: Accelerate DETR Training by Introducing Query DeNoisingcited by 486
  13. 2021Less is More: ClipBERT for Video-and-Language Learning via Sparse Samplingcited by 484
  14. 2020End-to-End Video Instance Segmentation with Transformerscited by 443
  15. 2022InternVideo: General Video Foundation Models via Generative and Discriminative Learningcited by 441
  16. 2021Conditional DETR for Fast Training Convergencecited by 422
  17. 2020Multi-modal Transformer for Video Retrievalcited by 395
  18. 2022Not All Patches are What You Need: Expediting Vision Transformers via Token Reorganizationscited by 359
  19. 2021MERLOT: Multimodal Neural Script Knowledge Modelscited by 311
  20. 2022MultiMAE: Multi-modal Multi-task Masked Autoencoderscited by 287
  21. 2022X-CLIP: End-to-End Multi-grained Contrastive Learning for Video-Text Retrievalcited by 279
  22. 2020Just Ask: Learning to Answer Questions from Millions of Narrated Videoscited by 248
  23. 2022Zero-Shot Video Question Answering via Frozen Bidirectional Language Modelscited by 229
  24. 2021AdaViT: Adaptive Vision Transformers for Efficient Image Recognitioncited by 228
  25. 2022Open-Vocabulary DETR with Conditional Matchingcited by 220
  26. 2023Token Merging for Fast Stable Diffusioncited by 220
  27. 2023Unmasked Teacher: Towards Training-Efficient Video Foundation Modelscited by 199
  28. 2021VIOLET : End-to-End Video-Language Transformers with Masked Visual-token Modelingcited by 187
  29. 2021Adaptive Token Sampling For Efficient Vision Transformerscited by 181
  30. 2022Frozen CLIP Models are Efficient Video Learnerscited by 181
  31. 2016Movie Descriptioncited by 180
  32. 2022All in One: Exploring Unified Video-Language Pre-trainingcited by 171
  33. 2021SPViT: Enabling Faster Vision Transformers via Soft Token Pruningcited by 165
  34. 2021SwinBERT: End-to-End Transformers with Sparse Attention for Video Captioningcited by 161
  35. 2021Align and Prompt: Video-and-Language Pre-training with Entity Promptscited by 149
  36. 2022OmniVL:One Foundation Model for Image-Language and Video-Language Taskscited by 143
  37. 2021Learned Token Pruning for Transformerscited by 138
  38. 2022Revealing Single Frame Bias for Video-and-Language Learningcited by 131

50 most cited of 62 All 62 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 8; 2025: 20; 2026: 13.

41 papers.

Sort:

By area:

  1. 2024LongVU: Spatiotemporal Adaptive Compression for Long Video-Language Understandingcited by 341
  2. 2024NVILA: Efficient Frontier Visual Language Modelscited by 206
  3. 2025FastVGGT: Training-Free Acceleration of Visual Geometry Transformercited by 93
  4. 2024Gramian Multimodal Representation Learning and Alignmentcited by 52
  5. 2025Multi-Granular Spatio-Temporal Token Merging for Training-Free Acceleration of Video LLMscited by 30
  6. 2025StreamAgent: Towards Anticipatory Agents for Streaming Video Understandingcited by 29
  7. 2024Importance-Based Token Merging for Efficient Image and Video Generationcited by 20
  8. 2025Principled Multimodal Representation Learningcited by 19
  9. 2025Keyframe-oriented Vision Token Pruning: Enhancing Efficiency of Large Vision Language Models on Long-Form Video Processingcited by 18
  10. 2024EfficientViM: Efficient Vision Mamba with Hidden State Mixer based State Space Dualitycited by 17
  11. 2024Leveraging Temporal Contextualization for Video Action Recognitioncited by 9
  12. 2024Principles of Visual Tokens for Efficient Video Understandingcited by 7
29 more
  1. 2025One Trajectory, One Token: Grounded Video Tokenization via Panoptic Sub-object Trajectorycited by 7
  2. 2025Make Your Training Flexible: Towards Deployment-Efficient Video Modelscited by 6
  3. 2025Bootstrap Your Own Views: Masked Ego-Exo Modeling for Fine-grained View-invariant Video Representationscited by 6
  4. 2025D'ej`a Vu: Efficient Video-Language Query Engine with Learning-based Inter-Frame Computation Reusecited by 5
  5. 2025Representation Shift: Unifying Token Compression with FlashAttentioncited by 5
  6. 2025D3ToM: Decider-Guided Dynamic Token Merging for Accelerating Diffusion MLLMscited by 5
  7. 2026ToProVAR: Efficient Visual Autoregressive Modeling via Tri-Dimensional Entropy-Aware Semantic Analysis and Sparsity Optimizationcited by 5
  8. 2025Language-Guided Temporal Token Pruning for Efficient VideoLLM Processingcited by 4
  9. 2026One Token per Highly Selective Frame: Towards Extreme Compression for Long Video Understandingcited by 4
  10. 2026DocPrune:Efficient Document Question Answering via Background, Question, and Comprehension-aware Token Pruningcited by 4
  11. 2025Diversity-Guided MLP Reduction for Efficient Large Vision Transformerscited by 3
  12. 2025TabFlash: Efficient Table Understanding with Progressive Question Conditioning and Token Focusingcited by 3
  13. 2026PyramidalWan: On Making Pretrained Video Model Pyramidal for Efficient Inferencecited by 3
  14. 2025Video, How Do Your Tokens Merge?cited by 2
  15. 2025BeltCrack: the First Sequential-image Industrial Conveyor Belt Crack Detection Dataset and Its Baseline with Triple-domain Feature Learningcited by 2
  16. 2026Compression Tells Intelligence: Visual Coding, Visual Token Technology, and the Unificationcited by 2
  17. 2026TrajTok: Learning Trajectory Tokens enables better Video Understandingcited by 2
  18. 2026EFlow: Fast Few-Step Video Generator Training from Scratch via Efficient Solution Flowcited by 2
  19. 2026DepthCache: Depth-Guided Training-Free Visual Token Merging for Vision-Language-Action Model Inferencecited by 1
  20. 2026LookWhen? Fast Video Recognition by Learning When, Where, and What to Computecited by 1
  21. 2026ReVision: Scaling Computer-Use Agents via Temporal Visual Redundancy Reductioncited by 1
  22. 2024MPT-PAR:Mix-Parameters Transformer for Panoramic Activity Recognitioncited by 0
  23. 2025CubistMerge: Spatial-Preserving Token Merging For Diverse ViT Backbonescited by 0
  24. 2025PSTTS: A Plug-and-Play Token Selector for Efficient Event-based Spatio-temporal Representation Learningcited by 0
  25. 2025Token Merging via Spatiotemporal Information Mining for Surgical Video Understandingcited by 0
  26. 2025Neighbor-Aware Token Reduction via Hilbert Curve for Vision Transformerscited by 0
  27. 2026Efficient Multi-View 3D Object Detection by Dynamic Token Selection and Fine-Tuningcited by 0
  28. 2026StoryVideoQA: Scaling Deep Video Understanding with a Large-Scale, Multi-Genre and Auto-Generated Datasetcited by 0
  29. 2026Text-Vision Synergistic Token Caching: A Training-Free Framework for Efficient Vision-Language-Action Inferencecited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Video Token Merging for Long-form Video Understanding
  2. 2024Vote&Mix: Plug-and-Play Token Reduction for Efficient Vision Transformer
  3. 2024Learning to Merge Tokens via Decoupled Embedding for Efficient Vision Transformers
  4. 2023VidToMe: Video Token Merging for Zero-Shot Video Editing
  5. 2024HaltingVT: Adaptive Token Halting Transformer for Efficient Video Recognition
  6. 2023Token Fusion: Bridging the Gap between Token Pruning and Token Merging
  7. 2026Video Patch Pruning: Efficient Video Instance Segmentation via Early Token Reduction
  8. 2026MergeOver: Post-Training Token Merging for Recursive Vision Transformers