mlsanityData to 14 Oct 2026

VideoCoCa: Video-Text Modeling with Zero-Shot Transfer from Contrastive Captioners

arXiv 2212.04979 ↗First posted 9 Dec 2022cs.CV cs.LG cs.MMcited by 70

Not on the map. Why?

Abstract (arXiv)

We explore an efficient approach to establish a foundational video-text model. We present VideoCoCa that maximally reuses a pretrained image-text contrastive captioner (CoCa) model and adapt it to video-text tasks with minimal extra training. While previous works adapt image-text models with various cross-frame fusion modules, we find that the generative attentional pooling and contrastive attentional pooling layers in CoCa are instantly adaptable to flattened frame embeddings, yielding state-of-the-art results on zero-shot video classification and zero-shot text-to-video retrieval. Furthermore, we explore lightweight finetuning on top of VideoCoCa, and achieve strong results on video question-answering and video captioning.

Key facts. Cited by 70 papers in the map (data through 14 Oct 2026). Builds on 66 of its 76 references that are in the map. First posted on arXiv 9 Dec 2022.

Builds on

66 of 76 references.

References · 66

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  4. 2015Rethinking the Inception Architecture for Computer Visioncited by 5,475
  5. 2017mixup: Beyond Empirical Risk Minimizationcited by 5,183
  6. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,386
  7. 2021On the Opportunities and Risks of Foundation Modelscited by 4,033
  8. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,864
  9. 2017Quo Vadis, Action Recognition? A New Model and the Kinetics Datasetcited by 3,270
  10. 2017Non-local Neural Networkscited by 2,293
  11. 2019RandAugment: Practical automated data augmentation with a reduced search spacecited by 2,269
  12. 2014CIDEr: Consensus-based Image Description Evaluationcited by 2,046
38 more
  1. 2017The Kinetics Human Action Video Datasetcited by 1,909
  2. 2021Is Space-Time Attention All You Need for Video Understanding?cited by 1,620
  3. 2021ViViT: A Video Vision Transformercited by 1,620
  4. 2022CoCa: Contrastive Captioners are Image-Text Foundation Modelscited by 1,274
  5. 2021Perceiver: General Perception with Iterative Attentioncited by 1,104
  6. 2021Video Swin Transformercited by 1,091
  7. 2016Deep Networks with Stochastic Depthcited by 979
  8. 2019HowTo100M: Learning a Text-Video Embedding by Watching Hundred Million Narrated Video Clipscited by 950
  9. 2018Adafactor: Adaptive Learning Rates with Sublinear Memory Costcited by 905
  10. 2021Florence: A New Foundation Model for Computer Visioncited by 804
  11. 2017Dense-Captioning Events in Videoscited by 793
  12. 2022PaLI: A Jointly-Scaled Multilingual Language-Image Modelcited by 766
  13. 2021CLIP4Clip: An Empirical Study of CLIP for End to End Video Clip Retrievalcited by 651
  14. 2021SimVLM: Simple Visual Language Model Pretraining with Weak Supervisioncited by 647
  15. 2017Towards Automatic Learning of Procedures from Web Instructional Videoscited by 619
  16. 2021VideoCLIP: Contrastive Pre-training for Zero-shot Video-Text Understandingcited by 602
  17. 2021LiT: Zero-Shot Transfer with Locked-image text Tuningcited by 577
  18. 2019ActivityNet-QA: A Dataset for Understanding Complex Web Videos via Question Answeringcited by 561
  19. 2016Hollywood in Homes: Crowdsourcing Data Collection for Activity Understandingcited by 537
  20. 2022GIT: A Generative Image-to-text Transformer for Vision and Languagecited by 537
  21. 2017Rethinking Spatiotemporal Feature Learning: Speed-Accuracy Trade-offs in Video Classificationcited by 521
  22. 2022Socratic Models: Composing Zero-Shot Multimodal Reasoning with Languagecited by 513
  23. 2022Image as a Foreign Language: BEiT Pretraining for All Vision and Vision-Language Taskscited by 511
  24. 2022InternVideo: General Video Foundation Models via Generative and Discriminative Learningcited by 447
  25. 2019VATEX: A Large-Scale, High-Quality Multilingual Dataset for Video-and-Language Researchcited by 414
  26. 2022Expanding Language-Image Pretrained Models for General Video Recognitioncited by 321
  27. 2021ActionCLIP: A New Paradigm for Video Action Recognitioncited by 320
  28. 2021Prompting Visual-Language Models for Efficient Video Understandingcited by 310
  29. 2020UniVL: A Unified Video and Language Pre-Training Model for Multimodal Understanding and Generationcited by 259
  30. 2020Just Ask: Learning to Answer Questions from Millions of Narrated Videoscited by 248
  31. 2022Zero-Shot Video Question Answering via Frozen Bidirectional Language Modelscited by 229
  32. 2021CLIP2Video: Mastering Video-Text Retrieval via Image CLIPcited by 225
  33. 2022UniFormer: Unifying Convolution and Self-attention for Visual Recognitioncited by 206
  34. 2021VIOLET : End-to-End Video-Language Transformers with Masked Visual-token Modelingcited by 188
  35. 2022Frozen CLIP Models are Efficient Video Learnerscited by 182
  36. 2020Fine-grained Video-Text Retrieval with Hierarchical Graph Reasoningcited by 180
  37. 2022All in One: Exploring Unified Video-Language Pre-trainingcited by 171
  38. 2022Multiview Transformers for Video Recognitioncited by 148

50 most cited of 66 All 66 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 11; 2024: 26; 2025: 22; 2026: 11.

70 papers.

Sort:

By area:

  1. 2023mPLUG-2: A Modularized Multi-modal Foundation Model Across Text, Image and Videocited by 154
  2. 2024Tarsier: Recipes for Training and Evaluating Large Video Description Modelscited by 141
  3. 2024AuroraCap: Efficient, Performant Video Detailed Captioning and a New Benchmarkcited by 140
  4. 2025VLM2Vec-V2: Advancing Multimodal Embedding for Videos, Images, and Visual Documentscited by 116
  5. 2024VideoPrism: A Foundational Visual Encoder for Video Understandingcited by 105
  6. 2024Advancing Multimodal Medical Capabilities of Geminicited by 90
  7. 2024TRACE: Temporal Grounding Video LLM via Causal Event Modelingcited by 89
  8. 2025Tarsier2: Advancing Large Vision-Language Models from Detailed Video Description to Comprehensive Video Understandingcited by 81
  9. 2024MoReVQA: Exploring Modular Reasoning Models for Video Question Answeringcited by 74
  10. 2024Exploring the Frontier of Vision-Language Models: A Survey of Current Methodologies and Future Directionscited by 68
  11. 2024Learning to Localize Objects Improves Spatial Reasoning in Visual-LLMscited by 49
  12. 2024EgoVideo: Exploring Egocentric Foundation Model and Downstream Adaptationcited by 44
38 more
  1. 2024Towards Generalist Robot Learning from Internet Video: A Surveycited by 40
  2. 2023Bootstrapping Vision-Language Learning with Decoupled Language Pre-trainingcited by 28
  3. 2024Understanding Long Videos with Multimodal Language Modelscited by 25
  4. 2023MaMMUT: A Simple Architecture for Joint Learning for MultiModal Taskscited by 24
  5. 2025UGC-VideoCaptioner: An Omni UGC Video Detail Caption Model and New Benchmarkscited by 19
  6. 2025Principled Multimodal Representation Learningcited by 19
  7. 2025A TRIANGLE Enables Multimodal Alignment Beyond Cosine Similaritycited by 18
  8. 2024TrafficVLM: A Controllable Visual Language Model for Traffic Video Captioningcited by 17
  9. 2023Expedited Training of Visual Conditioned Language Generation via Redundancy Reductioncited by 15
  10. 2023Mirasol3B: A Multimodal Autoregressive model for time-aligned and contextual modalitiescited by 15
  11. 2023VideoGLUE: Video General Understanding Evaluation of Foundation Modelscited by 14
  12. 2024Enhancing Long Video Understanding via Hierarchical Event-Based Memorycited by 14
  13. 2024Scaling Robot Policy Learning via Zero-Shot Labeling with Foundation Modelscited by 14
  14. 2023Video-Teller: Enhancing Cross-Modal Generation with Fusion and Decouplingcited by 12
  15. 2024VideoEval: Comprehensive Benchmark Suite for Low-Cost Evaluation of Video Foundation Modelcited by 12
  16. 20253D Foundation Model for Generalizable Disease Detection in Head Computed Tomographycited by 12
  17. 2025OwlCap: Harmonizing Motion-Detail for Video Captioning via HMD-270K and Caption Set Equivalence Rewardcited by 12
  18. 2025Pretrained Image-Text Models are Secretly Video Captionerscited by 11
  19. 2025VideoComp: Advancing Fine-Grained Compositional and Temporal Alignment in Video-Text Modelscited by 11
  20. 2025Divide, then Ground: Adapting Frame Selection to Query Types for Long-Form Video Understandingcited by 11
  21. 2024TOPA: Extending Large Language Models for Video Understanding via Text-Only Pre-Alignmentcited by 10
  22. 2025Learning Streaming Video Representation via Multitask Trainingcited by 10
  23. 2026Attend Before Attention: Efficient and Scalable Video Understanding via Autoregressive Gazingcited by 9
  24. 2023Actor-agnostic Multi-label Action Recognition with Multi-modal Querycited by 8
  25. 2025Fine-Grained Captioning of Long Videos through Scene Graph Consolidationcited by 8
  26. 2024AdaCM2: On Understanding Extremely Long-Term Video with Adaptive Cross-Modality Memory Reductioncited by 7
  27. 2024Video DataFlywheel: Resolving the Impossible Data Trinity in Video-Language Understandingcited by 6
  28. 2024M2-RAAP: A Multi-Modal Recipe for Advancing Adaptation-based Pre-training towards Effective and Efficient Zero-shot Video-text Retrievalcited by 5
  29. 2024VidLA: Video-Language Alignment at Scalecited by 5
  30. 2024FocusChat: Text-guided Long Video Understanding via Spatiotemporal Information Filteringcited by 5
  31. 2025ViQAgent: Zero-Shot Video Question Answering via Agent with Open-Vocabulary Grounding Validationcited by 5
  32. 2026Closing the Modality Gap Aligns Group-Wise Semanticscited by 5
  33. 2023IMAD: IMage-Augmented multi-modal Dialoguecited by 4
  34. 2024SRL-CLIP: Efficient CLIP Video Adaptation via Structured Semantic Role Labelscited by 4
  35. 2024Health AI Developer Foundationscited by 4
  36. 2025Bidirectional Likelihood Estimation with Multi-Modal Large Language Models for Text-Video Retrievalcited by 4
  37. 2025Bridging Vision Language Models and Symbolic Grounding for Video Question Answeringcited by 4
  38. 2025Learning Skill-Attributes for Transferable Assessment in Videocited by 4

50 most cited of 70 All 70 (JSON)