mlsanityData to 14 Oct 2026

VideoPrism: A Foundational Visual Encoder for Video Understanding

arXiv 2402.13217 ↗First posted 20 Feb 2024cs.CV cs.AIcited by 105

Not on the map. Why?

Abstract (arXiv)

We introduce VideoPrism, a general-purpose video encoder that tackles diverse video understanding tasks with a single frozen model. We pretrain VideoPrism on a heterogeneous corpus containing 36M high-quality video-caption pairs and 582M video clips with noisy parallel text (e.g., ASR transcripts). The pretraining approach improves upon masked autoencoding by global-local distillation of semantic video embeddings and a token shuffling scheme, enabling VideoPrism to focus primarily on the video modality while leveraging the invaluable text associated with videos. We extensively test VideoPrism on four broad groups of video understanding tasks, from web video question answering to CV for science, achieving state-of-the-art performance on 31 out of 33 video understanding benchmarks. Our models are released at https://github.com/google-deepmind/videoprism.

Key facts. Cited by 105 papers in the map (data through 14 Oct 2026). Builds on 121 of its 144 references that are in the map. First posted on arXiv 20 Feb 2024.

Builds on

121 of 144 references.

References · 121

  1. 2017Attention Is All You Needcited by 60,361
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  3. 2020Language Models are Few-Shot Learnerscited by 32,419
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  5. 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,639
  6. 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,974
  7. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,403
  8. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,386
  9. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,322
  10. 2021On the Opportunities and Risks of Foundation Modelscited by 4,033
  11. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,864
  12. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,516
38 more
  1. 2021BEiT: BERT Pre-Training of Image Transformerscited by 2,375
  2. 2017The Kinetics Human Action Video Datasetcited by 1,909
  3. 2018SlowFast Networks for Video Recognitioncited by 1,867
  4. 2021ViViT: A Video Vision Transformercited by 1,620
  5. 2021Ego4D: Around the World in 3,000 Hours of Egocentric Videocited by 1,556
  6. 2023Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understandingcited by 1,471
  7. 2023Video-LLaVA: Learning United Visual Representation by Alignment Before Projectioncited by 1,402
  8. 2016Unsupervised Learning of Visual Representations by Solving Jigsaw Puzzlescited by 1,382
  9. 2022VideoMAE: Masked Autoencoders are Data-Efficient Learners for Self-Supervised Video Pre-Trainingcited by 1,322
  10. 2022CoCa: Contrastive Captioners are Image-Text Foundation Modelscited by 1,274
  11. 2021Frozen in Time: A Joint Video and Image Encoder for End-to-End Retrievalcited by 1,177
  12. 2023ImageBind: One Embedding Space To Bind Them Allcited by 1,175
  13. 2023PaLM 2 Technical Reportcited by 1,162
  14. 2017The "something something" video database for learning and evaluating visual common sensecited by 1,103
  15. 2023Video-ChatGPT: Towards Detailed Video Understanding via Large Vision and Language Modelscited by 1,062
  16. 2021Scaling Vision Transformerscited by 999
  17. 2019HowTo100M: Learning a Text-Video Embedding by Watching Hundred Million Narrated Video Clipscited by 950
  18. 2018Adafactor: Adaptive Learning Rates with Sublinear Memory Costcited by 905
  19. 2021iBOT: Image BERT Pre-Training with Online Tokenizercited by 903
  20. 2021Florence: A New Foundation Model for Computer Visioncited by 804
  21. 2017Dense-Captioning Events in Videoscited by 793
  22. 2022PaLI: A Jointly-Scaled Multilingual Language-Image Modelcited by 766
  23. 2021NExT-QA:Next Phase of Question-Answering to Explaining Temporal Actionscited by 744
  24. 2022EVA: Exploring the Limits of Masked Visual Representation Learning at Scalecited by 691
  25. 2021CLIP4Clip: An Empirical Study of CLIP for End to End Video Clip Retrievalcited by 651
  26. 2021FLAVA: A Foundational Language And Vision Alignment Modelcited by 649
  27. 2017Towards Automatic Learning of Procedures from Web Instructional Videoscited by 619
  28. 2022AudioLM: a Language Modeling Approach to Audio Generationcited by 616
  29. 2021VideoCLIP: Contrastive Pre-training for Zero-shot Video-Text Understandingcited by 602
  30. 2017TALL: Temporal Activity Localization via Language Querycited by 584
  31. 2021Masked Feature Prediction for Self-Supervised Visual Pre-Trainingcited by 576
  32. 2023LLaMA-VID: An Image is Worth 2 Tokens in Large Language Modelscited by 546
  33. 2020Rescaling Egocentric Visioncited by 540
  34. 2016Hollywood in Homes: Crowdsourcing Data Collection for Activity Understandingcited by 537
  35. 2023VideoMAE V2: Scaling Video Masked Autoencoders with Dual Maskingcited by 528
  36. 2022Socratic Models: Composing Zero-Shot Multimodal Reasoning with Languagecited by 513
  37. 2023InternVid: A Large-scale Video-Text Dataset for Multimodal Understanding and Generationcited by 497
  38. 2017AVA: A Video Dataset of Spatio-temporally Localized Atomic Visual Actionscited by 449

50 most cited of 121 All 121 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 22; 2025: 43; 2026: 40.

105 papers.

Sort:

By area:

  1. 2025V-JEPA 2: Self-Supervised Video Models Enable Understanding, Prediction and Planningcited by 809
  2. 2025Perception Encoder: The best visual embeddings are not at the output of the networkcited by 326
  3. 2024An Introduction to Vision-Language Modelingcited by 178
  4. 2024Tarsier: Recipes for Training and Evaluating Large Video Description Modelscited by 141
  5. 2026V-JEPA 2.1: Unlocking Dense Features in Video Self-Supervised Learningcited by 117
  6. 2024TRACE: Temporal Grounding Video LLM via Causal Event Modelingcited by 89
  7. 2024VCA: Video Curious Agent for Long Video Understandingcited by 56
  8. 2024Gramian Multimodal Representation Learning and Alignmentcited by 54
  9. 2024Foundation Models for Video Understanding: A Surveycited by 46
  10. 2024Towards Generalist Robot Learning from Internet Video: A Surveycited by 40
  11. 2025A Survey on Knowledge-Oriented Retrieval-Augmented Generationcited by 36
  12. 2024Scaling 4D Representationscited by 30
38 more
  1. 2025AToken: A Unified Tokenizer for Visioncited by 30
  2. 2024Task Preference Optimization: Improving Multimodal Large Language Models with Vision Task Alignmentcited by 29
  3. 2024EgoCVR: An Egocentric Benchmark for Fine-Grained Composed Video Retrievalcited by 21
  4. 2025Principled Multimodal Representation Learningcited by 19
  5. 2025A TRIANGLE Enables Multimodal Alignment Beyond Cosine Similaritycited by 18
  6. 2024TrafficVLM: A Controllable Visual Language Model for Traffic Video Captioningcited by 17
  7. 2025From Image to Video: An Empirical Study of Diffusion Representationscited by 15
  8. 2025Pushing the Frontier of Audiovisual Perception with Large-Scale Multimodal Correspondence Learningcited by 13
  9. 2024VideoEval: Comprehensive Benchmark Suite for Low-Cost Evaluation of Video Foundation Modelcited by 12
  10. 2025Rethinking JEPA: Compute-Efficient Video SSL with Frozen Teacherscited by 12
  11. 2025VideoComp: Advancing Fine-Grained Compositional and Temporal Alignment in Video-Text Modelscited by 11
  12. 2025VideoMiner: Iteratively Grounding Key Frames of Hour-Long Videos via Tree-based Group Relative Policy Optimizationcited by 11
  13. 2024TOPA: Extending Large Language Models for Video Understanding via Text-Only Pre-Alignmentcited by 10
  14. 2025Unifying Specialized Visual Encoders for Video Language Modelscited by 10
  15. 2025Learning Streaming Video Representation via Multitask Trainingcited by 10
  16. 2025Causality Matters: How Temporal Information Emerges in Video Language Modelscited by 10
  17. 2025SMILE: Infusing Spatial and Motion Semantics in Masked Video Learningcited by 9
  18. 2025VideoExpert: Augmented LLM for Temporal-Sensitive Video Understandingcited by 9
  19. 2025SurgBench: A Unified Large-Scale Benchmark for Surgical Video Analysiscited by 9
  20. 2025PrismAudio: Decomposed Chain-of-Thoughts and Multi-dimensional Rewards for Video-to-Audio Generationcited by 9
  21. 2025Fine-Grained Captioning of Long Videos through Scene Graph Consolidationcited by 8
  22. 2025One Trajectory, One Token: Grounded Video Tokenization via Panoptic Sub-object Trajectorycited by 7
  23. 2025FrameOracle: Learning What to See and How Much to See in Videoscited by 7
  24. 2025InternVideo-Next: Towards General Video Foundation Models without Video-Text Supervisioncited by 7
  25. 2024Localizing Events in Videos with Multimodal Queriescited by 6
  26. 2024Learning to Localize Actions in Instructional Videos with LLM-Based Multi-Pathway Text-Video Alignmentcited by 6
  27. 2025SciVid: Cross-Domain Evaluation of Video Models in Scientific Applicationscited by 6
  28. 2024AlphaChimp: Tracking and Behavior Recognition of Chimpanzeescited by 5
  29. 2024TRecViT: A Recurrent Video Transformercited by 5
  30. 2025Animal behavioral analysis and neural encoding with transformer-based self-supervised pretrainingcited by 5
  31. 2025Dynamic Reflections: Probing Video Representations with Text Alignmentcited by 5
  32. 2024LocoMotion: Learning Motion-Focused Video-Language Representationscited by 4
  33. 2025SEVERE++: Evaluating Benchmark Sensitivity in Generalization of Video Representation Learningcited by 4
  34. 2025Frozen Forecasting: A Unified Evaluationcited by 4
  35. 2025Bidirectional Likelihood Estimation with Multi-Modal Large Language Models for Text-Video Retrievalcited by 4
  36. 2026PyraTok: Language-Aligned Pyramidal Tokenizer for Video Understanding and Generationcited by 4
  37. 2026VidVec: Unlocking Video MLLM Embeddings for Video-Text Retrievalcited by 4
  38. 2026Towards Uniformity and Alignment for Multimodal Representation Learningcited by 4

50 most cited of 105 All 105 (JSON)