mlsanityData to 7 Oct 2026

Less is More: ClipBERT for Video-and-Language Learning via Sparse Sampling

arXiv 2102.06183 ↗First posted 11 Feb 2021cs.CV cs.CLcited by 484

Not on the map. Why?

Cited by 57 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

The canonical approach to video-and-language learning (e.g., video question answering) dictates a neural model to learn from offline-extracted dense video features from vision models and text features from language models. These feature extractors are trained independently and usually on tasks different from the target domains, rendering these fixed features sub-optimal for downstream tasks. Moreover, due to the high computational overload of dense video features, it is often difficult (or infeasible) to plug feature extractors directly into existing approaches for easy finetuning. To provide a remedy to this dilemma, we propose a generic framework ClipBERT that enables affordable end-to-end learning for video-and-language tasks, by employing sparse sampling, where only a single or a few sparsely sampled short clips from a video are used at each training step. Experiments on text-to-video retrieval and video question answering on six datasets demonstrate that ClipBERT outperforms (or is on par with) existing methods that exploit full-length videos, suggesting that end-to-end learning with just a few sparsely sampled clips is often more accurate than using densely extracted offline features from full-length videos, proving the proverbial less-is-more principle. Videos in the datasets are from considerably different domains and lengths, ranging from 3-second generic domain GIF videos to 180-second YouTube human activity videos, showing the generalization ability of our approach. Comprehensive ablation studies and thorough analyses are provided to dissect what factors lead to this success. Our code is publicly available at https://github.com/jayleicn/ClipBERT

Key facts. Cited by 484 papers in the map (data through 7 Oct 2026). Builds on 71 of its 87 references that are in the map. First posted on arXiv 11 Feb 2021.

Builds on

71 of 87 references.

References · 71

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  4. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  5. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
  6. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  7. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  8. 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,617
  9. 2014Going Deeper with Convolutionscited by 4,231
  10. 2013Distributed Representations of Words and Phrases and their Compositionalitycited by 3,462
  11. 2017Quo Vadis, Action Recognition? A New Model and the Kinetics Datasetcited by 3,259
  12. 2019XLNet: Generalized Autoregressive Pretraining for Language Understandingcited by 2,801
38 more
  1. 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,772
  2. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,646
  3. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,498
  4. 2015VQA: Visual Question Answeringcited by 2,489
  5. 2017Non-local Neural Networkscited by 2,290
  6. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,219
  7. 2019HuggingFace's Transformers: State-of-the-art Natural Language Processingcited by 2,201
  8. 2017The Kinetics Human Action Video Datasetcited by 1,894
  9. 2018SlowFast Networks for Video Recognitioncited by 1,862
  10. 2019LXMERT: Learning Cross-Modality Encoder Representations from Transformerscited by 1,548
  11. 2016Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translationcited by 1,471
  12. 2015Microsoft COCO Captions: Data Collection and Evaluation Servercited by 1,447
  13. 2017Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answeringcited by 1,310
  14. 2020Oscar: Object-Semantics Aligned Pre-training for Vision-Language Taskscited by 1,213
  15. 2017A Closer Look at Spatiotemporal Convolutions for Action Recognitioncited by 1,139
  16. 2014Two-Stream Convolutional Networks for Action Recognition in Videoscited by 1,067
  17. 2015Aligning Books and Movies: Towards Story-like Visual Explanations by Watching Movies and Reading Bookscited by 1,041
  18. 2019VL-BERT: Pre-training of Generic Visual-Linguistic Representationscited by 954
  19. 2019HowTo100M: Learning a Text-Video Embedding by Watching Hundred Million Narrated Video Clipscited by 946
  20. 2016Temporal Segment Networks: Towards Good Practices for Deep Action Recognitioncited by 899
  21. 2018VizWiz Grand Challenge: Answering Visual Questions from Blind Peoplecited by 852
  22. 2017Dense-Captioning Events in Videoscited by 789
  23. 2019VideoBERT: A Joint Model for Video and Language Representation Learningcited by 702
  24. 2017Localizing Moments in Video with Natural Languagecited by 660
  25. 2020X3D: Expanding Architectures for Efficient Video Recognitioncited by 635
  26. 2017Towards Automatic Learning of Procedures from Web Instructional Videoscited by 616
  27. 2017TALL: Temporal Activity Localization via Language Querycited by 581
  28. 2017Rethinking Spatiotemporal Feature Learning: Speed-Accuracy Trade-offs in Video Classificationcited by 521
  29. 2019End-to-End Learning of Visual Representations from Uncurated Instructional Videoscited by 506
  30. 2019Unified Vision-Language Pre-Training for Image Captioning and VQAcited by 494
  31. 2018TVQA: Localized, Compositional Video Question Answeringcited by 463
  32. 2019Unicoder-VL: A Universal Encoder for Vision and Language by Cross-modal Pre-trainingcited by 451
  33. 2021Unifying Vision-and-Language Tasks via Text Generationcited by 411
  34. 2020Multi-modal Transformer for Video Retrievalcited by 395
  35. 2017Learning Spatio-Temporal Representation with Pseudo-3D Residual Networkscited by 386
  36. 2017TGIF-QA: Toward Spatio-Temporal Reasoning in Visual Question Answeringcited by 378
  37. 2020HERO: Hierarchical Encoder for Video+Language Omni-representation Pre-trainingcited by 362
  38. 2020Large-Scale Adversarial Training for Vision-and-Language Representation Learningcited by 327

50 most cited of 71 All 71 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 54; 2022: 112; 2023: 138; 2024: 94; 2025: 58; 2026: 28.

484 papers.

Sort:

By area:

  1. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,310
  2. 2021Frozen in Time: A Joint Video and Image Encoder for End-to-End Retrievalcited by 1,175
  3. 2021CLIP4Clip: An Empirical Study of CLIP for End to End Video Clip Retrievalcited by 645
  4. 2021VideoCLIP: Contrastive Pre-training for Zero-shot Video-Text Understandingcited by 596
  5. 2024LLaVA-Video: Video Instruction Tuning With Synthetic Datacited by 550
  6. 2022GIT: A Generative Image-to-text Transformer for Vision and Languagecited by 536
  7. 2021DenseCLIP: Language-Guided Dense Prediction with Context-Aware Promptingcited by 530
  8. 2022InternVideo: General Video Foundation Models via Generative and Discriminative Learningcited by 441
  9. 2022Multimodal Learning with Transformers: A Surveycited by 399
  10. 2024Panda-70M: Captioning 70M Videos with Multiple Cross-Modality Teacherscited by 389
  11. 2023An Embodied Generalist Agent in 3D Worldcited by 355
  12. 2021LAVT: Language-Aware Vision Transformer for Referring Image Segmentationcited by 347
38 more
  1. 2021ScanQA: 3D Question Answering for Spatial Scene Understandingcited by 340
  2. 2024VideoAgent: Long-form Video Understanding with Large Language Model as Agentcited by 323
  3. 2021ActionCLIP: A New Paradigm for Video Action Recognitioncited by 317
  4. 2021MERLOT: Multimodal Neural Script Knowledge Modelscited by 311
  5. 2021Prompting Visual-Language Models for Efficient Video Understandingcited by 309
  6. 2022SQA3D: Situated Question Answering in 3D Scenescited by 303
  7. 2022X-CLIP: End-to-End Multi-grained Contrastive Learning for Video-Text Retrievalcited by 279
  8. 2023Vid2Seq: Large-Scale Pretraining of a Visual Language Model for Dense Video Captioningcited by 263
  9. 2022Egocentric Video-Language Pretrainingcited by 231
  10. 2022Zero-Shot Video Question Answering via Frozen Bidirectional Language Modelscited by 229
  11. 2021Advancing High-Resolution Video-Language Representation with Large-Scale Video Transcriptionscited by 226
  12. 2021CLIP2Video: Mastering Video-Text Retrieval via Image CLIPcited by 223
  13. 2024VideoLLM-online: Online Video Large Language Model for Streaming Videocited by 216
  14. 2024STAR: A Benchmark for Situated Reasoning in Real-World Videoscited by 213
  15. 2022Learning Video Representations from Large Language Modelscited by 204
  16. 2023Self-Chained Image-Language Model for Video Localization and Question Answeringcited by 200
  17. 2023Unmasked Teacher: Towards Training-Efficient Video Foundation Modelscited by 199
  18. 2021Scaling Up Vision-Language Pre-training for Image Captioningcited by 196
  19. 2023UniVTG: Towards Unified Video-Language Temporal Groundingcited by 189
  20. 2021VIOLET : End-to-End Video-Language Transformers with Masked Visual-token Modelingcited by 187
  21. 2022MERLOT Reserve: Neural Script Knowledge through Vision and Language and Soundcited by 184
  22. 2024LLaVA-3D: A Simple yet Effective Pathway to Empowering LMMs with 3D-awarenesscited by 178
  23. 2023A Simple LLM Framework for Long-Range Video Question-Answeringcited by 176
  24. 2022All in One: Exploring Unified Video-Language Pre-trainingcited by 171
  25. 2024Scene-LLM: Extending Language Model for 3D Visual Understanding and Reasoningcited by 170
  26. 2023VAST: A Vision-Audio-Subtitle-Text Omni-Modality Foundation Model and Datasetcited by 168
  27. 2022Revisiting the "Video" in Video-Language Understandingcited by 167
  28. 2021SwinBERT: End-to-End Transformers with Sparse Attention for Video Captioningcited by 161
  29. 2021Align and Prompt: Video-and-Language Pre-training with Entity Promptscited by 149
  30. 2024LongVLM: Efficient Long Video Understanding via Large Language Modelscited by 146
  31. 2022OmniVL:One Foundation Model for Image-Language and Video-Language Taskscited by 143
  32. 2023VALOR: Vision-Audio-Language Omni-Perception Pretraining Model and Datasetcited by 138
  33. 2023Can I Trust Your Answer? Visually Grounded Video Question Answeringcited by 133
  34. 2022Revealing Single Frame Bias for Video-and-Language Learningcited by 131
  35. 2023EgoVLPv2: Egocentric Video-Language Pre-training with Fusion in the Backbonecited by 130
  36. 2022X-Pool: Cross-Modal Language-Video Attention for Text-Video Retrievalcited by 127
  37. 2022Language Models with Image Descriptors are Strong Few-Shot Video-Language Learnerscited by 117
  38. 2022Vision-Language Pre-training: Basics, Recent Advances, and Future Trendscited by 117

50 most cited of 484 All 484 (JSON)