mlsanityData to 7 Oct 2026

Revealing Single Frame Bias for Video-and-Language Learning

arXiv 2206.03428 ↗First posted 7 Jun 2022cs.CV cs.AI cs.CLcited by 131

Not on the map. Why?

Cited by 26 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Training an effective video-and-language model intuitively requires multiple frames as model inputs. However, it is unclear whether using multiple frames is beneficial to downstream tasks, and if yes, whether the performance gain is worth the drastically-increased computation and memory costs resulting from using more frames. In this work, we explore single-frame models for video-and-language learning. On a diverse set of video-and-language tasks (including text-to-video retrieval and video question answering), we show the surprising result that, with large-scale pre-training and a proper frame ensemble strategy at inference time, a single-frame trained model that does not consider temporal information can achieve better performance than existing methods that use multiple frames for training. This result reveals the existence of a strong "static appearance bias" in popular video-and-language datasets. Therefore, to allow for a more comprehensive evaluation of video-and-language models, we propose two new retrieval tasks based on existing fine-grained action recognition datasets that encourage temporal modeling. Our code is available at https://github.com/jayleicn/singularity

Key facts. Cited by 131 papers in the map (data through 7 Oct 2026). Builds on 60 of its 72 references that are in the map. First posted on arXiv 7 Jun 2022.

Builds on

60 of 72 references.

References · 60

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  5. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  6. 2016SGDR: Stochastic Gradient Descent with Warm Restartscited by 4,993
  7. 2020Training data-efficient image transformers & distillation through attentioncited by 4,560
  8. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,310
  9. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
  10. 2017Quo Vadis, Action Recognition? A New Model and the Kinetics Datasetcited by 3,259
  11. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,646
  12. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,498
38 more
  1. 2015VQA: Visual Question Answeringcited by 2,489
  2. 2021BEiT: BERT Pre-Training of Image Transformerscited by 2,360
  3. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,219
  4. 2017The Kinetics Human Action Video Datasetcited by 1,894
  5. 2021Align before Fuse: Vision and Language Representation Learning with Momentum Distillationcited by 1,612
  6. 2021Is Space-Time Attention All You Need for Video Understanding?cited by 1,609
  7. 2019LXMERT: Learning Cross-Modality Encoder Representations from Transformerscited by 1,548
  8. 2015Microsoft COCO Captions: Data Collection and Evaluation Servercited by 1,447
  9. 2017Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answeringcited by 1,310
  10. 2019VisualBERT: A Simple and Performant Baseline for Vision and Languagecited by 1,291
  11. 2021ViLT: Vision-and-Language Transformer Without Convolution or Region Supervisioncited by 1,224
  12. 2020Oscar: Object-Semantics Aligned Pre-training for Vision-Language Taskscited by 1,213
  13. 2021Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Conceptscited by 1,209
  14. 2021Frozen in Time: A Joint Video and Image Encoder for End-to-End Retrievalcited by 1,175
  15. 2017The "something something" video database for learning and evaluating visual common sensecited by 1,092
  16. 2021Perceiver: General Perception with Iterative Attentioncited by 1,089
  17. 2019HowTo100M: Learning a Text-Video Embedding by Watching Hundred Million Narrated Video Clipscited by 946
  18. 2016Temporal Segment Networks: Towards Good Practices for Deep Action Recognitioncited by 899
  19. 2017Dense-Captioning Events in Videoscited by 789
  20. 2019VideoBERT: A Joint Model for Video and Language Representation Learningcited by 702
  21. 2017Localizing Moments in Video with Natural Languagecited by 660
  22. 2018Annotation Artifacts in Natural Language Inference Datacited by 656
  23. 2021CLIP4Clip: An Empirical Study of CLIP for End to End Video Clip Retrievalcited by 645
  24. 2021Perceiver IO: A General Architecture for Structured Inputs & Outputscited by 616
  25. 2021VideoCLIP: Contrastive Pre-training for Zero-shot Video-Text Understandingcited by 596
  26. 2019ActivityNet-QA: A Dataset for Understanding Complex Web Videos via Question Answeringcited by 559
  27. 2018From Recognition to Cognition: Visual Commonsense Reasoningcited by 553
  28. 2021Less is More: ClipBERT for Video-and-Language Learning via Sparse Samplingcited by 484
  29. 2018TVQA: Localized, Compositional Video Question Answeringcited by 463
  30. 2021Unifying Vision-and-Language Tasks via Text Generationcited by 411
  31. 2020Multi-modal Transformer for Video Retrievalcited by 395
  32. 2017TGIF-QA: Toward Spatio-Temporal Reasoning in Visual Question Answeringcited by 378
  33. 2019Using Pre-Training Can Improve Model Robustness and Uncertaintycited by 373
  34. 2020HERO: Hierarchical Encoder for Video+Language Omni-representation Pre-trainingcited by 362
  35. 2021MERLOT: Multimodal Neural Script Knowledge Modelscited by 311
  36. 2020ActBERT: Learning Global-Local Video-Text Representationscited by 267
  37. 2020Just Ask: Learning to Answer Questions from Millions of Narrated Videoscited by 248
  38. 2020UNIMO: Towards Unified-Modal Understanding and Generation via Cross-Modal Contrastive Learningcited by 239

50 most cited of 60 All 60 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 12; 2023: 44; 2024: 33; 2025: 26; 2026: 16.

131 papers.

Sort:

By area:

  1. 2024LLaVA-Video: Video Instruction Tuning With Synthetic Datacited by 550
  2. 2023InternVid: A Large-scale Video-Text Dataset for Multimodal Understanding and Generationcited by 488
  3. 2024VideoMamba: State Space Model for Efficient Video Understandingcited by 325
  4. 2024MA-LMM: Memory-Augmented Large Multimodal Model for Long-Term Video Understandingcited by 223
  5. 2023Video Understanding with Large Language Models: A Surveycited by 203
  6. 2023Self-Chained Image-Language Model for Video Localization and Question Answeringcited by 200
  7. 2023Unmasked Teacher: Towards Training-Efficient Video Foundation Modelscited by 199
  8. 2023VAST: A Vision-Audio-Subtitle-Text Omni-Modality Foundation Model and Datasetcited by 168
  9. 2023mPLUG-2: A Modularized Multi-modal Foundation Model Across Text, Image and Videocited by 154
  10. 2024Tarsier: Recipes for Training and Evaluating Large Video Description Modelscited by 140
  11. 2023VALOR: Vision-Audio-Language Omni-Perception Pretraining Model and Datasetcited by 138
  12. 2023Can I Trust Your Answer? Visually Grounded Video Question Answeringcited by 133
38 more
  1. 2022Vision-Language Pre-training: Basics, Recent Advances, and Future Trendscited by 117
  2. 2024VideoPrism: A Foundational Visual Encoder for Video Understandingcited by 103
  3. 2024TemporalBench: Benchmarking Fine-grained Temporal Understanding for Multimodal Video Modelscited by 96
  4. 2024VideoHallucer: Evaluating Intrinsic and Extrinsic Hallucinations in Large Video-Language Modelscited by 88
  5. 2022VindLU: A Recipe for Effective Video-and-Language Pretrainingcited by 87
  6. 2023Verbs in Action: Improving verb understanding in video-language modelscited by 76
  7. 2023Audio-Visual LLM for Video Understandingcited by 76
  8. 2024MoReVQA: Exploring Modular Reasoning Models for Video Question Answeringcited by 74
  9. 2022LAVENDER: Unifying Video-Language Understanding as Masked Language Modelingcited by 73
  10. 2022HiTeA: Hierarchical Temporal-Aware Video-Language Pre-trainingcited by 72
  11. 2024TOMATO: Assessing Visual Temporal Reasoning Capabilities in Multimodal Foundation Modelscited by 71
  12. 2022VideoCoCa: Video-Text Modeling with Zero-Shot Transfer from Contrastive Captionerscited by 69
  13. 2023HierVL: Learning Hierarchical Video-Language Embeddingscited by 66
  14. 2023VITATECS: A Diagnostic Dataset for Temporal Concept Understanding of Video-Language Modelscited by 55
  15. 2023Test of Time: Instilling Video-Language Models with a Sense of Timecited by 51
  16. 2023Unified Coarse-to-Fine Alignment for Video-Text Retrievalcited by 49
  17. 2023Paxion: Patching Action Knowledge in Video-Language Foundation Modelscited by 42
  18. 2023BT-Adapter: Video Conversation is Feasible Without Video Instruction Tuningcited by 41
  19. 2024vid-TLDR: Training Free Token merging for Light-weight Video Transformercited by 41
  20. 2023AutoEval-Video: An Automatic Benchmark for Assessing Large Vision Language Models in Open-Ended Video Question Answeringcited by 40
  21. 2024Vinoground: Scrutinizing LMMs over Dense Temporal Reasoning with Short Videoscited by 37
  22. 2023A Simple Recipe for Contrastively Pre-training Video-First Encoders Beyond 16 Framescited by 34
  23. 2025A Shortcut-aware Video-QA Benchmark for Physical Understanding via Minimal Video Pairscited by 30
  24. 2025PAI-Bench: A Comprehensive Benchmark For Physical AIcited by 30
  25. 2023VideoCon: Robust Video-Language Alignment via Contrast Captionscited by 29
  26. 2024VideoQA in the Era of LLMs: An Empirical Studycited by 29
  27. 2025Video-RTS: Rethinking Reinforcement Learning and Test-Time Scaling for Efficient and Enhanced Video Reasoningcited by 29
  28. 2022Composing Ensembles of Pre-trained Models via Iterative Consensuscited by 28
  29. 2025Seeing the Arrow of Time in Large Multimodal Modelscited by 28
  30. 2023ViLMA: A Zero-Shot Benchmark for Linguistic and Temporal Grounding in Video-Language Modelscited by 26
  31. 2022Zero-Shot Video Captioning with Evolving Pseudo-Tokenscited by 25
  32. 2024LVD-2M: A Long-take Video Dataset with Temporally Dense Captionscited by 24
  33. 2025FAVOR-Bench: A Comprehensive Benchmark for Fine-Grained Video Motion Understandingcited by 24
  34. 2023Reading Between the Lanes: Text VideoQA on the Roadcited by 23
  35. 2023Building an Open-Vocabulary Video CLIP Model with Better Architectures, Optimization and Datacited by 21
  36. 2022Watching the News: Towards VideoQA Models that can Readcited by 20
  37. 2023SPOT! Revisiting Video-Language Models for Event Understandingcited by 20
  38. 2024VideoLLaMB: Long Streaming Video Understanding with Recurrent Memory Bridgescited by 20

50 most cited of 131 All 131 (JSON)