mlsanityData to 7 Oct 2026

ActBERT: Learning Global-Local Video-Text Representations

arXiv 2011.07231 ↗First posted 14 Nov 2020cs.CVcited by 267

Not on the map. Why?

Abstract (arXiv)

In this paper, we introduce ActBERT for self-supervised learning of joint video-text representations from unlabeled data. First, we leverage global action information to catalyze the mutual interactions between linguistic texts and local regional objects. It uncovers global and local visual clues from paired video sequences and text descriptions for detailed visual and text relation modeling. Second, we introduce an ENtangled Transformer block (ENT) to encode three sources of information, i.e., global actions, local regional objects, and linguistic descriptions. Global-local correspondences are discovered via judicious clues extraction from contextual information. It enforces the joint videotext representation to be aware of fine-grained objects as well as global human intention. We validate the generalization capability of ActBERT on downstream video-and language tasks, i.e., text-video clip retrieval, video captioning, video question answering, action segmentation, and action step localization. ActBERT significantly outperforms the state-of-the-arts, demonstrating its superiority in video-text representation learning.

Key facts. Cited by 267 papers in the map (data through 7 Oct 2026). First posted on arXiv 14 Nov 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 60; 2022: 94; 2023: 67; 2024: 32; 2025: 10; 2026: 4.

267 papers.

Sort:

By area:

  1. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  2. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,310
  3. 2021Frozen in Time: A Joint Video and Image Encoder for End-to-End Retrievalcited by 1,175
  4. 2023VideoChat: Chat-Centric Video Understandingcited by 851
  5. 2021NExT-QA:Next Phase of Question-Answering to Explaining Temporal Actionscited by 735
  6. 2021CLIP4Clip: An Empirical Study of CLIP for End to End Video Clip Retrievalcited by 645
  7. 2021VideoCLIP: Contrastive Pre-training for Zero-shot Video-Text Understandingcited by 596
  8. 2022GIT: A Generative Image-to-text Transformer for Vision and Languagecited by 536
  9. 2023MovieChat: From Dense Token to Sparse Memory for Long Video Understandingcited by 536
  10. 2023InternVid: A Large-scale Video-Text Dataset for Multimodal Understanding and Generationcited by 488
  11. 2021Less is More: ClipBERT for Video-and-Language Learning via Sparse Samplingcited by 484
  12. 2022MineDojo: Building Open-Ended Embodied Agents with Internet-Scale Knowledgecited by 469
38 more
  1. 2022InternVideo: General Video Foundation Models via Generative and Discriminative Learningcited by 441
  2. 2021Unifying Vision-and-Language Tasks via Text Generationcited by 411
  3. 2022Multimodal Learning with Transformers: A Surveycited by 399
  4. 2023LanguageBind: Extending Video-Language Pretraining to N-modality by Language-based Semantic Alignmentcited by 366
  5. 2022Expanding Language-Image Pretrained Models for General Video Recognitioncited by 320
  6. 2021MERLOT: Multimodal Neural Script Knowledge Modelscited by 311
  7. 2023A Survey on Multimodal Large Language Models for Autonomous Drivingcited by 288
  8. 2022X-CLIP: End-to-End Multi-grained Contrastive Learning for Video-Text Retrievalcited by 279
  9. 2022Egocentric Video-Language Pretrainingcited by 231
  10. 2022Zero-Shot Video Question Answering via Frozen Bidirectional Language Modelscited by 229
  11. 2021Advancing High-Resolution Video-Language Representation with Large-Scale Video Transcriptionscited by 226
  12. 2021Keeping Your Eye on the Ball: Trajectory Attention in Video Transformerscited by 224
  13. 2021CLIP2Video: Mastering Video-Text Retrieval via Image CLIPcited by 223
  14. 2022Learning Video Representations from Large Language Modelscited by 204
  15. 2023Video Understanding with Large Language Models: A Surveycited by 203
  16. 2021Scaling Up Vision-Language Pre-training for Image Captioningcited by 196
  17. 2021VIOLET : End-to-End Video-Language Transformers with Masked Visual-token Modelingcited by 187
  18. 2022All in One: Exploring Unified Video-Language Pre-trainingcited by 171
  19. 2022Revisiting the "Video" in Video-Language Understandingcited by 167
  20. 2021SwinBERT: End-to-End Transformers with Sparse Attention for Video Captioningcited by 161
  21. 2022VLP: A Survey on Vision-Language Pre-trainingcited by 159
  22. 2023Large-scale Multi-Modal Pre-trained Models: A Comprehensive Surveycited by 153
  23. 2021End-to-end Temporal Action Detection with Transformercited by 150
  24. 2021Align and Prompt: Video-and-Language Pre-training with Entity Promptscited by 149
  25. 2022OmniVL:One Foundation Model for Image-Language and Video-Language Taskscited by 143
  26. 2022Revealing Single Frame Bias for Video-and-Language Learningcited by 131
  27. 2023EgoVLPv2: Egocentric Video-Language Pre-training with Fusion in the Backbonecited by 130
  28. 2022X-Pool: Cross-Modal Language-Video Attention for Text-Video Retrievalcited by 127
  29. 2021Improving Video-Text Retrieval by Multi-Stream Corpus Alignment and Dual Softmax Losscited by 116
  30. 2022End-to-end Generative Pretraining for Multimodal Video Captioningcited by 115
  31. 2021T2VLAD: Global-Local Sequence Alignment for Text-Video Retrievalcited by 103
  32. 2021VLM: Task-agnostic Video-Language Model Pre-training for Video Understandingcited by 102
  33. 2021TACo: Token-aware Cascade Contrastive Learning for Video-Text Alignmentcited by 102
  34. 2022TubeDETR: Spatio-Temporal Video Grounding with Transformerscited by 97
  35. 2021VALUE: A Multi-Task Benchmark for Video-and-Language Understanding Evaluationcited by 95
  36. 2021Thinking Fast and Slow: Efficient Text-to-Visual Retrieval with Transformerscited by 93
  37. 2021HiT: Hierarchical Transformer with Momentum Contrast for Video-Text Retrievalcited by 87
  38. 2022VindLU: A Recipe for Effective Video-and-Language Pretrainingcited by 87

50 most cited of 267 All 267 (JSON)