mlsanityData to 7 Oct 2026

Frozen in Time: A Joint Video and Image Encoder for End-to-End Retrieval

arXiv 2104.00650 ↗First posted 1 Apr 2021cs.CVcited by 1,175

Not on the map. Why?

Cited by 223 in Diffusion models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Our objective in this work is video-text retrieval - in particular a joint embedding that enables efficient text-to-video retrieval. The challenges in this area include the design of the visual architecture and the nature of the training data, in that the available large scale video-text training datasets, such as HowTo100M, are noisy and hence competitive performance is achieved only at scale through large amounts of compute. We address both these challenges in this paper. We propose an end-to-end trainable model that is designed to take advantage of both large-scale image and video captioning datasets. Our model is an adaptation and extension of the recent ViT and Timesformer architectures, and consists of attention in both space and time. The model is flexible and can be trained on both image and video text datasets, either independently or in conjunction. It is trained with a curriculum learning schedule that begins by treating images as 'frozen' snapshots of video, and then gradually learns to attend to increasing temporal context when trained on video datasets. We also provide a new video-text pretraining dataset WebVid-2M, comprised of over two million videos with weak captions scraped from the internet. Despite training on datasets that are an order of magnitude smaller, we show that this approach yields state-of-the-art results on standard downstream video-retrieval benchmarks including MSR-VTT, MSVD, DiDeMo and LSMDC.

Key facts. Cited by 1,175 papers in the map (data through 7 Oct 2026). Builds on 68 of its 82 references that are in the map. First posted on arXiv 1 Apr 2021.

Builds on

68 of 82 references.

References · 68

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  5. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  6. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  7. 2020End-to-End Object Detection with Transformerscited by 7,384
  8. 2020Training data-efficient image transformers & distillation through attentioncited by 4,560
  9. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,243
  10. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
  11. 2017Quo Vadis, Action Recognition? A New Model and the Kinetics Datasetcited by 3,259
  12. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,646
38 more
  1. 2015VQA: Visual Question Answeringcited by 2,489
  2. 2017Non-local Neural Networkscited by 2,290
  3. 2017The Kinetics Human Action Video Datasetcited by 1,894
  4. 2021Is Space-Time Attention All You Need for Video Understanding?cited by 1,609
  5. 2019Generating Long Sequences with Sparse Transformerscited by 1,523
  6. 2015Microsoft COCO Captions: Data Collection and Evaluation Servercited by 1,447
  7. 2020Oscar: Object-Semantics Aligned Pre-training for Vision-Language Taskscited by 1,213
  8. 2021Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Conceptscited by 1,209
  9. 2017A Closer Look at Spatiotemporal Convolutions for Action Recognitioncited by 1,139
  10. 2019HowTo100M: Learning a Text-Video Embedding by Watching Hundred Million Narrated Video Clipscited by 946
  11. 2017Dense-Captioning Events in Videoscited by 789
  12. 2018Image Transformercited by 770
  13. 2017Localizing Moments in Video with Natural Languagecited by 660
  14. 2017Towards Automatic Learning of Procedures from Web Instructional Videoscited by 616
  15. 2017Can Spatiotemporal 3D CNNs Retrace the History of 2D CNNs and ImageNet?cited by 551
  16. 2019Stand-Alone Self-Attention in Vision Modelscited by 542
  17. 2016Hollywood in Homes: Crowdsourcing Data Collection for Activity Understandingcited by 535
  18. 2017Rethinking Spatiotemporal Feature Learning: Speed-Accuracy Trade-offs in Video Classificationcited by 521
  19. 2019End-to-End Learning of Visual Representations from Uncurated Instructional Videoscited by 506
  20. 2021Less is More: ClipBERT for Video-and-Language Learning via Sparse Samplingcited by 484
  21. 2018TVQA: Localized, Compositional Video Question Answeringcited by 463
  22. 2018Stacked Cross Attention for Image-Text Matchingcited by 414
  23. 2020Multi-modal Transformer for Video Retrievalcited by 395
  24. 2020HERO: Hierarchical Encoder for Video+Language Omni-representation Pre-trainingcited by 362
  25. 2017Relation Networks for Object Detectioncited by 305
  26. 2019On the Relationship between Self-Attention and Convolutional Layerscited by 299
  27. 2017Temporal Segment Networks for Action Recognition in Videoscited by 279
  28. 2020Self-Supervised MultiModal Versatile Networkscited by 276
  29. 2020ActBERT: Learning Global-Local Video-Text Representationscited by 267
  30. 2016Image Captioning with Semantic Attentioncited by 260
  31. 2020UniVL: A Unified Video and Language Pre-Training Model for Multimodal Understanding and Generationcited by 258
  32. 2018End-to-End Dense Video Captioning with Masked Transformercited by 214
  33. 2019Use What You Have: Video Retrieval Using Representations From Collaborative Expertscited by 214
  34. 2018A Joint Sequence Fusion Model for Video Question Answering and Retrievalcited by 208
  35. 2014Unifying Visual-Semantic Embeddings with Multimodal Neural Language Modelscited by 197
  36. 2015A Dataset for Movie Descriptioncited by 195
  37. 2016Movie Descriptioncited by 180
  38. 2020Support-set bottlenecks for video-text representation learningcited by 170

50 most cited of 68 All 68 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 30; 2022: 115; 2023: 270; 2024: 362; 2025: 277; 2026: 121.

1,175 papers.

Sort:

By area:

  1. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  2. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,310
  3. 2023Stable Video Diffusion: Scaling Latent Video Diffusion Models to Large Datasetscited by 2,526
  4. 2024CogVideoX: Text-to-Video Diffusion Models with An Expert Transformercited by 2,198
  5. 2023VBench: Comprehensive Benchmark Suite for Video Generative Modelscited by 1,755
  6. 2022Make-A-Video: Text-to-Video Generation without Text-Video Datacited by 1,725
  7. 2023Align your Latents: High-Resolution Video Synthesis with Latent Diffusion Modelscited by 1,465
  8. 2023Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understandingcited by 1,461
  9. 2023AnimateDiff: Animate Your Personalized Text-to-Image Diffusion Models without Specific Tuningcited by 1,418
  10. 2023Video-LLaVA: Learning United Visual Representation by Alignment Before Projectioncited by 1,385
  11. 2023ImageBind: One Embedding Space To Bind Them Allcited by 1,162
  12. 2023MVBench: A Comprehensive Multi-modal Video Understanding Benchmarkcited by 1,086
38 more
  1. 2022Tune-A-Video: One-Shot Tuning of Image Diffusion Models for Text-to-Video Generationcited by 958
  2. 2024WorldSimBench: Towards Video Generation Models as World Simulatorscited by 867
  3. 2023VideoChat: Chat-Centric Video Understandingcited by 851
  4. 2021Florence: A New Foundation Model for Computer Visioncited by 802
  5. 2024VideoLLaMA 2: Advancing Spatial-Temporal Modeling and Audio Understanding in Video-LLMscited by 749
  6. 2024Genie: Generative Interactive Environmentscited by 706
  7. 2023ModelScope Text-to-Video Technical Reportcited by 651
  8. 2021CLIP4Clip: An Empirical Study of CLIP for End to End Video Clip Retrievalcited by 645
  9. 2023Instruction Tuning for Large Language Models: A Surveycited by 617
  10. 2023NExT-GPT: Any-to-Any Multimodal LLMcited by 610
  11. 2023MotionCtrl: A Unified and Flexible Motion Controller for Video Generationcited by 568
  12. 2024VideoCrafter2: Overcoming Data Limitations for High-Quality Video Diffusion Modelscited by 568
  13. 2024LLaVA-Video: Video Instruction Tuning With Synthetic Datacited by 550
  14. 2023LLaMA-VID: An Image is Worth 2 Tokens in Large Language Modelscited by 539
  15. 2023MovieChat: From Dense Token to Sparse Memory for Long Video Understandingcited by 536
  16. 2024Latte: Latent Diffusion Transformer for Video Generationcited by 536
  17. 2023VideoCrafter1: Open Diffusion Models for High-Quality Video Generationcited by 530
  18. 2023VideoMAE V2: Scaling Video Masked Autoencoders with Dual Maskingcited by 516
  19. 2022Socratic Models: Composing Zero-Shot Multimodal Reasoning with Languagecited by 513
  20. 2023InternVid: A Large-scale Video-Text Dataset for Multimodal Understanding and Generationcited by 488
  21. 2022TEMOS: Generating diverse human motions from textual descriptionscited by 486
  22. 2022Phenaki: Variable Length Video Generation From Open Domain Textual Descriptioncited by 475
  23. 2022MagicVideo: Efficient Video Generation With Latent Diffusion Modelscited by 463
  24. 2023DynamiCrafter: Animating Open-domain Images with Video Diffusion Priorscited by 463
  25. 2023VideoComposer: Compositional Video Synthesis with Motion Controllabilitycited by 461
  26. 2023DriveGPT4: Interpretable End-to-end Autonomous Driving via Large Language Modelcited by 455
  27. 2024CameraCtrl: Enabling Camera Control for Text-to-Video Generationcited by 448
  28. 2023Generative Multimodal Models are In-Context Learnerscited by 446
  29. 2022InternVideo: General Video Foundation Models via Generative and Discriminative Learningcited by 441
  30. 2023Chat-UniVi: Unified Visual Representation Empowers Large Language Models with Image and Video Understandingcited by 368
  31. 2021Attention Bottlenecks for Multimodal Fusioncited by 361
  32. 2024Pyramidal Flow Matching for Efficient Video Generative Modelingcited by 354
  33. 2024Retrieval-Augmented Generation for AI-Generated Content: A Surveycited by 352
  34. 2024TempCompass: Do Video LLMs Really Understand Videos?cited by 352
  35. 2023LAVIE: High-Quality Video Generation with Cascaded Latent Diffusion Modelscited by 349
  36. 2023I2VGen-XL: High-Quality Image-to-Video Synthesis via Cascaded Diffusion Modelscited by 343
  37. 2024OpenVid-1M: A Large-Scale High-Quality Dataset for Text-to-video Generationcited by 332
  38. 2023VideoFusion: Decomposed Diffusion Models for High-Quality Video Generationcited by 330

50 most cited of 1,175 All 1,175 (JSON)