mlsanityData to 14 Oct 2026

Don't Look Twice: Faster Video Transformers with Run-Length Tokenization

arXiv 2411.05222 ↗First posted 7 Nov 2024cs.CV cs.LGcited by 43

Not on the map. Why?

Abstract (arXiv)

Transformers are slow to train on videos due to extremely large numbers of input tokens, even though many video tokens are repeated over time. Existing methods to remove such uninformative tokens either have significant overhead, negating any speedup, or require tuning for different datasets and examples. We present Run-Length Tokenization (RLT), a simple approach to speed up video transformers inspired by run-length encoding for data compression. RLT efficiently finds and removes runs of patches that are repeated over time prior to model inference, then replaces them with a single patch and a positional encoding to represent the resulting token's new length. Our method is content-aware, requiring no tuning for different datasets, and fast, incurring negligible overhead. RLT yields a large speedup in training, reducing the wall-clock time to fine-tune a video transformer by 30% while matching baseline model performance. RLT also works without any training, increasing model throughput by 35% with only 0.1% drop in accuracy. RLT speeds up training at 30 FPS by more than 100%, and on longer video datasets, can reduce the token count by up to 80%. Our project page is at https://rccchoudhury.github.io/projects/rlt/.

Key facts. Cited by 43 papers in the map (data through 14 Oct 2026). Builds on 49 of its 56 references that are in the map. First posted on arXiv 7 Nov 2024.

Builds on

49 of 56 references.

References · 49

  1. 2017Attention Is All You Needcited by 60,361
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  3. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
  4. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,139
  5. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,403
  6. 2022Scalable Diffusion Models with Transformerscited by 6,054
  7. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,673
  8. 2020Taming Transformers for High-Resolution Image Synthesiscited by 3,253
  9. 2024SAM 2: Segment Anything in Images and Videoscited by 2,863
  10. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,598
  11. 2019RandAugment: Practical automated data augmentation with a reduced search spacecited by 2,269
  12. 2017The Kinetics Human Action Video Datasetcited by 1,909
37 more
  1. 2018SlowFast Networks for Video Recognitioncited by 1,867
  2. 2021Is Space-Time Attention All You Need for Video Understanding?cited by 1,620
  3. 2021ViViT: A Video Vision Transformercited by 1,620
  4. 2022VideoMAE: Masked Autoencoders are Data-Efficient Learners for Self-Supervised Video Pre-Trainingcited by 1,322
  5. 2017The "something something" video database for learning and evaluating visual common sensecited by 1,103
  6. 2014Two-Stream Convolutional Networks for Action Recognition in Videoscited by 1,068
  7. 2022Token Merging: Your ViT But Fastercited by 1,021
  8. 2021Multiscale Vision Transformerscited by 897
  9. 2024WorldSimBench: Towards Video Generation Models as World Simulatorscited by 881
  10. 2021DynamicViT: Efficient Vision Transformers with Dynamic Token Sparsificationcited by 864
  11. 2018TSM: Temporal Shift Module for Efficient Video Understandingcited by 856
  12. 2022GroupViT: Semantic Segmentation Emerges from Text Supervisioncited by 531
  13. 2023VideoMAE V2: Scaling Video Masked Autoencoders with Dual Maskingcited by 528
  14. 2021MViTv2: Improved Multiscale Vision Transformers for Classification and Detectioncited by 519
  15. 2021VATT: Transformers for Multimodal Self-Supervised Learning from Raw Video, Audio and Textcited by 429
  16. 2022Masked Autoencoders As Spatiotemporal Learnerscited by 429
  17. 2022Not All Patches are What You Need: Expediting Vision Transformers via Token Reorganizationscited by 363
  18. 2021AdaViT: Adaptive Tokens for Efficient Vision Transformercited by 362
  19. 2017Temporal Relational Reasoning in Videoscited by 324
  20. 2019COIN: A Large-scale Dataset for Comprehensive Instructional Video Analysiscited by 318
  21. 2022Scaling Language-Image Pre-training via Maskingcited by 317
  22. 2023Hiera: A Hierarchical Vision Transformer without the Bells-and-Whistlescited by 290
  23. 2023Patch n' Pack: NaViT, a Vision Transformer for any Aspect Ratio and Resolutioncited by 229
  24. 2021Multi-Scale Vision Longformer: A New Vision Transformer for High-Resolution Image Encodingcited by 207
  25. 2022MeMViT: Memory-Augmented Multiscale Vision Transformer for Efficient Long-Term Video Recognitioncited by 189
  26. 2021SPViT: Enabling Faster Vision Transformers via Soft Token Pruningcited by 167
  27. 2022FlexiViT: One Model for All Patch Sizescited by 152
  28. 2019SCSampler: Sampling Salient Clips from Video for Efficient Action Recognitioncited by 134
  29. 2021TokenLearner: What Can 8 Learned Tokens Do for Images and Videos?cited by 127
  30. 2018ECO: Efficient Convolutional Network for Online Video Understandingcited by 115
  31. 2017Compressed Video Action Recognitioncited by 106
  32. 2021Token Pooling in Vision Transformerscited by 69
  33. 2022Learning to Merge Tokens in Vision Transformerscited by 65
  34. 2022Extreme Masking for Learning Instance and Distributed Visual Representationscited by 21
  35. 2024STA-V2A: Video-to-Audio Generation with Semantic and Temporal Alignmentcited by 21
  36. 2023Training a Large Video Model on a Single Machine in a Daycited by 20
  37. 2022EVEREST: Efficient Masked Video Autoencoder by Removing Redundant Spatiotemporal Tokenscited by 15

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 2; 2025: 21; 2026: 20.

43 papers.

Sort:

By area:

  1. 2025VideoLLaMA 3: Frontier Multimodal Foundation Models for Image and Video Understandingcited by 518
  2. 2025TimeChat-Online: 80% Visual Tokens are Naturally Redundant in Streaming Videoscited by 97
  3. 2025Visual Planning: Let's Think Only with Imagescited by 85
  4. 2025Dynamic Chunking for End-to-End Hierarchical Sequence Modelingcited by 72
  5. 2025HoliTom: Holistic Token Merging for Fast Video Large Language Modelscited by 58
  6. 2026LLaVA-OneVision-2: Towards Next-Generation Perceptual Intelligencecited by 39
  7. 2025Shifting AI Efficiency From Model-Centric to Data-Centric Compressioncited by 29
  8. 2025Improving LLM Video Understanding with 16 Frames Per Secondcited by 26
  9. 2025OmniV-Med: Scaling Medical Vision-Language Model for Universal Visual Understandingcited by 17
  10. 2024AsymRnR: Video Diffusion Transformers Acceleration with Asymmetric Reduction and Restorationcited by 16
  11. 2025Multimodal Medical Code Tokenizercited by 14
  12. 2025Accelerating Vision Transformers with Adaptive Patch Sizescited by 13
31 more
  1. 2026OneVision-Encoder: Codec-Aligned Sparsity as a Foundational Principle for Multimodal Intelligencecited by 12
  2. 2025Causality Matters: How Temporal Information Emerges in Video Language Modelscited by 10
  3. 2025One Trajectory, One Token: Grounded Video Tokenization via Panoptic Sub-object Trajectorycited by 7
  4. 2026LongVideo-R1: Smart Navigation for Low-cost Long Video Understandingcited by 7
  5. 2025Bootstrap Your Own Views: Masked Ego-Exo Modeling for Fine-grained View-invariant Video Representationscited by 6
  6. 2026PyramidalWan: On Making Pretrained Video Model Pyramidal for Efficient Inferencecited by 4
  7. 2026PixelPrune: Pixel-Level Adaptive Visual Token Reduction via Predictive Codingcited by 4
  8. 2024Hydraulis: Balancing Large Transformer Model Training via Co-designing Parallel Strategies and Data Assignmentcited by 3
  9. 2025Top-Down Compression: Revisit Efficient Vision Token Projection for Visual Instruction Tuningcited by 3
  10. 2025LET-US: Long Event-Text Understanding of Scenescited by 3
  11. 2025Revisiting MLLM Token Technology through the Lens of Classical Visual Codingcited by 3
  12. 2026TrajTok: Learning Trajectory Tokens enables better Video Understandingcited by 3
  13. 2025Towards Fast, Memory-based and Data-Efficient Vision-Language Policycited by 2
  14. 2025QLIP: A Dynamic Quadtree Vision Prior Enhances MLLM Performance Without Retrainingcited by 2
  15. 2026Compression Tells Intelligence: Visual Coding, Visual Token Technology, and the Unificationcited by 2
  16. 2025Static or Dynamic: Towards Query-Adaptive Token Selection for Video Question Answeringcited by 1
  17. 2025VSI: Visual Subtitle Integration for Keyframe Selection to enhance Long Video Understandingcited by 1
  18. 2026Video Compression Meets Video Generation: Latent Inter-Frame Pruning with Attention Recoverycited by 1
  19. 2026Unified Spatio-Temporal Token Scoring for Efficient Video VLMscited by 1
  20. 2026LookWhen? Fast Video Recognition by Learning When, Where, and What to Computecited by 1
  21. 2026ReVision: Scaling Computer-Use Agents via Temporal Visual Redundancy Reductioncited by 1
  22. 2025StretchySnake: Flexible SSM Training Unlocks Action Recognition Across Spatio-Temporal Scalescited by 0
  23. 2026APB-V: Accelerating Long-Video Understanding via Sequence-Parallelism-aware Approximate Attentioncited by 0
  24. 2026Respiratory Status Detection with Video Transformerscited by 0
  25. 2026Learnable Motion-Focused Tokenization for Effective and Efficient Video Unsupervised Domain Adaptationcited by 0
  26. 2026EAST: Early Action Prediction Sampling Strategy with Token Maskingcited by 0
  27. 2026Latent Inter-Frame Pruning: A Training-Free Method Bridging Traditional Video Compression and Modern Diffusion Transformers for Efficient Generationcited by 0
  28. 2026Adaptive Tokenisation Via Temporal Redundancy Masking And Latent Inpaintingcited by 0
  29. 2026LongVU-TTT: Causal Test-Time Training for Visual Resampling in Long Video Understandingcited by 0
  30. 2026I Have a Stream: Making Self-Supervised Learning Work on Continuous Videocited by 0
  31. 2026Visual Tokens on Demand: Visual Budgeting for Scalable and Efficient Multi-Image Reasoning in Multimodal LLMscited by 0