mlsanityData to 7 Oct 2026

TESTA: Temporal-Spatial Token Aggregation for Long-form Video-Language Understanding

arXiv 2310.19060 ↗First posted 29 Oct 2023cs.CV cs.AI cs.CLcited by 49

Not on the map. Why?

Abstract (arXiv)

Large-scale video-language pre-training has made remarkable strides in advancing video-language understanding tasks. However, the heavy computational burden of video encoding remains a formidable efficiency bottleneck, particularly for long-form videos. These videos contain massive visual tokens due to their inherent 3D properties and spatiotemporal redundancy, making it challenging to capture complex temporal and spatial relationships. To tackle this issue, we propose an efficient method called TEmporal-Spatial Token Aggregation (TESTA). TESTA condenses video semantics by adaptively aggregating similar frames, as well as similar patches within each frame. TESTA can reduce the number of visual tokens by 75% and thus accelerate video encoding. Building upon TESTA, we introduce a pre-trained video-language model equipped with a divided space-time token aggregation module in each video encoder block. We evaluate our model on five datasets for paragraph-to-video retrieval and long-form VideoQA tasks. Experimental results show that TESTA improves computing efficiency by 1.7 times, and achieves significant performance gains from its scalability in processing longer input frames, e.g., +13.7 R@1 on QuerYD and +6.5 R@1 on Condensed Movie.

Key facts. Cited by 49 papers in the map (data through 7 Oct 2026). Builds on 51 of its 61 references that are in the map. First posted on arXiv 29 Oct 2023.

Builds on

51 of 61 references.

References · 51

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  5. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,482
  6. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,339
  7. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,310
  8. 2021Align before Fuse: Vision and Language Representation Learning with Momentum Distillationcited by 1,612
  9. 2021ViViT: A Video Vision Transformercited by 1,610
  10. 2021Is Space-Time Attention All You Need for Video Understanding?cited by 1,609
  11. 2022VideoMAE: Masked Autoencoders are Data-Efficient Learners for Self-Supervised Video Pre-Trainingcited by 1,305
  12. 2021Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Conceptscited by 1,209
38 more
  1. 2021Frozen in Time: A Joint Video and Image Encoder for End-to-End Retrievalcited by 1,175
  2. 2022Token Merging: Your ViT But Fastercited by 995
  3. 2021DynamicViT: Efficient Vision Transformers with Dynamic Token Sparsificationcited by 849
  4. 2017Dense-Captioning Events in Videoscited by 789
  5. 2017Localizing Moments in Video with Natural Languagecited by 660
  6. 2021CLIP4Clip: An Empirical Study of CLIP for End to End Video Clip Retrievalcited by 645
  7. 2021VideoCLIP: Contrastive Pre-training for Zero-shot Video-Text Understandingcited by 596
  8. 2019ActivityNet-QA: A Dataset for Understanding Complex Web Videos via Question Answeringcited by 559
  9. 2022GroupViT: Semantic Segmentation Emerges from Text Supervisioncited by 529
  10. 2021MViTv2: Improved Multiscale Vision Transformers for Classification and Detectioncited by 516
  11. 2021Less is More: ClipBERT for Video-and-Language Learning via Sparse Samplingcited by 484
  12. 2020Spatiotemporal Contrastive Video Representation Learningcited by 372
  13. 2021MERLOT: Multimodal Neural Script Knowledge Modelscited by 311
  14. 2022X-CLIP: End-to-End Multi-grained Contrastive Learning for Video-Text Retrievalcited by 279
  15. 2020Just Ask: Learning to Answer Questions from Millions of Narrated Videoscited by 248
  16. 2021Advancing High-Resolution Video-Language Representation with Large-Scale Video Transcriptionscited by 226
  17. 2021VIOLET : End-to-End Video-Language Transformers with Masked Visual-token Modelingcited by 187
  18. 2021Align and Prompt: Video-and-Language Pre-training with Entity Promptscited by 149
  19. 2021Towards Long-Form Video Understandingcited by 146
  20. 2022OmniVL:One Foundation Model for Image-Language and Video-Language Taskscited by 143
  21. 2022X-Pool: Cross-Modal Language-Video Attention for Text-Video Retrievalcited by 127
  22. 2017YouTube-BoundingBoxes: A Large High-Precision Human-Annotated Data Set for Object Detection in Videocited by 109
  23. 2018Learning a Text-Video Embedding from Incomplete and Heterogeneous Datacited by 103
  24. 2020Condensed Movies: Story Based Retrieval with Contextual Embeddingscited by 95
  25. 2022VindLU: A Recipe for Effective Video-and-Language Pretrainingcited by 87
  26. 2021TEACHTEXT: CrossModal Generalized Distillation for Text-Video Retrievalcited by 81
  27. 2022Disentangled Representation Learning for Text-Video Retrievalcited by 77
  28. 2022CLIP-ViP: Adapting Pre-trained Image-Text Model to Video-Language Representation Alignmentcited by 74
  29. 2022HiTeA: Hierarchical Temporal-Aware Video-Language Pre-trainingcited by 72
  30. 2021Token Pooling in Vision Transformerscited by 68
  31. 2022Long-Form Video-Language Pre-Training with Multimodal Temporal Contrastive Learningcited by 59
  32. 2022A CLIP-Hitchhiker's Guide to Long Video Retrievalcited by 57
  33. 2023CrossGET: Cross-Guided Ensemble of Tokens for Accelerating Vision-Language Transformerscited by 44
  34. 2023Prompt Pre-Training with Twenty-Thousand Classes for Open-Vocabulary Visual Recognitioncited by 35
  35. 2021Learning Relation Alignment for Calibrated Cross-modal Retrievalcited by 21
  36. 2022LiteVL: Efficient Video-Language Learning with Enhanced Spatial-Temporal Modelingcited by 16
  37. 2022Delving into the Openness of CLIPcited by 10
  38. 2019DCA: Diversified Co-Attention towards Informative Live Video Commentingcited by 8

50 most cited of 51 All 51 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 4; 2024: 18; 2025: 16; 2026: 11.

49 papers.

Sort:

By area:

  1. 2024Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysiscited by 1,375
  2. 2023TimeChat: A Time-sensitive Multimodal Large Language Model for Long Video Understandingcited by 399
  3. 2023Chat-UniVi: Unified Visual Representation Empowers Large Language Models with Image and Video Understandingcited by 368
  4. 2024TempCompass: Do Video LLMs Really Understand Videos?cited by 352
  5. 2024LongVU: Spatiotemporal Adaptive Compression for Long Video-Language Understandingcited by 341
  6. 2024MA-LMM: Memory-Augmented Large Multimodal Model for Long-Term Video Understandingcited by 223
  7. 2024AuroraCap: Efficient, Performant Video Detailed Captioning and a New Benchmarkcited by 137
  8. 2024DyCoke: Dynamic Compression of Tokens for Fast Video Large Language Modelscited by 93
  9. 2025TimeChat-Online: 80% Visual Tokens are Naturally Redundant in Streaming Videoscited by 90
  10. 2024DeCo: Decoupling Token Compression from Semantic Abstraction in Multimodal Large Language Modelscited by 74
  11. 2025HoliTom: Holistic Token Merging for Fast Video Large Language Modelscited by 58
  12. 2023VITATECS: A Diagnostic Dataset for Temporal Concept Understanding of Video-Language Modelscited by 55
37 more
  1. 2025Token Reduction Should Go Beyond Efficiency in Generative Models -- From Vision, Language to Multimodalitycited by 49
  2. 2025StreamingTOM: Streaming Token Compression for Efficient Video Understandingcited by 41
  3. 2024TempMe: Video Temporal Token Merging for Efficient Text-Video Retrievalcited by 39
  4. 2024xGen-MM-Vid (BLIP-3-Video): You Only Need 32 Tokens to Represent a Video Even in VLMscited by 34
  5. 2024ReTaKe: Reducing Temporal and Knowledge Redundancy for Long Video Understandingcited by 33
  6. 2024Enhancing Video-Language Representations with Structural Spatio-Temporal Alignmentcited by 32
  7. 2025Temporal Working Memory: Query-Guided Segment Refinement for Enhanced Multimodal Understandingcited by 28
  8. 2024Next Token Prediction Towards Multimodal Intelligence: A Comprehensive Surveycited by 27
  9. 2025Next Block Prediction: Video Generation via Semi-Autoregressive Modelingcited by 26
  10. 2024PVC: Progressive Visual Token Compression for Unified Image and Video Processing in Large Vision-Language Modelscited by 25
  11. 2024Video Token Merging for Long-form Video Understandingcited by 22
  12. 2024LinVT: Empower Your Image-level Large Language Model to Understand Videoscited by 19
  13. 2025HierarQ: Task-Aware Hierarchical Q-Former for Enhanced Video Understandingcited by 17
  14. 2024Learning Video Context as Interleaved Multimodal Sequencescited by 16
  15. 2025MammAlps: A multi-view video behavior monitoring dataset of wild mammals in the Swiss Alpscited by 14
  16. 2024Leveraging Temporal Contextualization for Video Action Recognitioncited by 9
  17. 2026Attend Before Attention: Efficient and Scalable Video Understanding via Autoregressive Gazingcited by 8
  18. 2026Unified Spatiotemporal Token Compression for Video-LLMs at Ultra-Low Retentioncited by 6
  19. 2025D'ej`a Vu: Efficient Video-Language Query Engine with Learning-based Inter-Frame Computation Reusecited by 5
  20. 2024Deploying Foundation Model Powered Agent Services: A Surveycited by 4
  21. 2025METok: Multi-Stage Event-based Token Compression for Efficient Long Video Understandingcited by 4
  22. 2026One Token per Highly Selective Frame: Towards Extreme Compression for Long Video Understandingcited by 4
  23. 2025TEMPLE: Incentivizing Temporal Understanding of Video Large Language Models via Progressive Pre-SFT Alignmentcited by 3
  24. 2026Learning Compact Video Representations for Efficient Long-form Video Understanding in Large Multimodal Modelscited by 3
  25. 2025REEF: Relevance-Aware and Efficient LLM Adapter for Video Understandingcited by 2
  26. 2025Video, How Do Your Tokens Merge?cited by 2
  27. 2025Free-MoRef: Instantly Multiplexing Context Perception Capabilities of Video-MLLMs within Single Inferencecited by 2
  28. 2026EgoSAT: A Comprehensive Benchmark of Egocentric Streaming Interaction Understandingcited by 2
  29. 2023A Video is Worth 10,000 Words: Training and Benchmarking with Diverse Captions for Better Long Video Retrievalcited by 1
  30. 2025Enhancing Sports Strategy with Video Analytics and Data Mining: Assessing the effectiveness of Multimodal LLMs in tennis video analysiscited by 1
  31. 2026Towards Temporal Compositional Reasoning in Long-Form Sports Videoscited by 1
  32. 2026LookWhen? Fast Video Recognition by Learning When, Where, and What to Computecited by 1
  33. 2025Token Merging via Spatiotemporal Information Mining for Surgical Video Understandingcited by 0
  34. 2026SAVE: Speech-Aware Video Representation Learning for Video-Text Retrievalcited by 0
  35. 2026Event-Aware Instructed Assistant for Referring Video Segmentationcited by 0
  36. 2026PCA: Persistence-Aware Compression and Aggregation for Fast Video Large Language Modelscited by 0
  37. 2026KeyRec: Bounded Visual Memory for Streaming and Long-Video Understandingcited by 0