mlsanityData to 7 Oct 2026

Support-set bottlenecks for video-text representation learning

arXiv 2010.02824 ↗First posted 6 Oct 2020cs.CVcited by 170

Not on the map. Why?

Abstract (arXiv)

The dominant paradigm for learning video-text representations -- noise contrastive learning -- increases the similarity of the representations of pairs of samples that are known to be related, such as text and video from the same sample, and pushes away the representations of all other pairs. We posit that this last behaviour is too strict, enforcing dissimilar representations even for samples that are semantically-related -- for example, visually similar videos or ones that share the same depicted action. In this paper, we propose a novel method that alleviates this by leveraging a generative model to naturally push these related samples together: each sample's caption must be reconstructed as a weighted combination of other support samples' visual representations. This simple idea ensures that representations are not overly-specialized to individual samples, are reusable across the dataset, and results in representations that explicitly encode semantics shared between samples, unlike noise contrastive learning. Our proposed method outperforms others by a large margin on MSR-VTT, VATEX and ActivityNet, and MSVD for video-to-text and text-to-video retrieval.

Key facts. Cited by 170 papers in the map (data through 7 Oct 2026). First posted on arXiv 6 Oct 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 37; 2022: 65; 2023: 39; 2024: 18; 2025: 6; 2026: 5.

170 papers.

Sort:

By area:

  1. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,310
  2. 2021Frozen in Time: A Joint Video and Image Encoder for End-to-End Retrievalcited by 1,175
  3. 2023ImageBind: One Embedding Space To Bind Them Allcited by 1,162
  4. 2021CLIP4Clip: An Empirical Study of CLIP for End to End Video Clip Retrievalcited by 645
  5. 2021VideoCLIP: Contrastive Pre-training for Zero-shot Video-Text Understandingcited by 596
  6. 2022GIT: A Generative Image-to-text Transformer for Vision and Languagecited by 536
  7. 2022Socratic Models: Composing Zero-Shot Multimodal Reasoning with Languagecited by 513
  8. 2022Multimodal Learning with Transformers: A Surveycited by 399
  9. 2022X-CLIP: End-to-End Multi-grained Contrastive Learning for Video-Text Retrievalcited by 279
  10. 2022Egocentric Video-Language Pretrainingcited by 231
  11. 2021Advancing High-Resolution Video-Language Representation with Large-Scale Video Transcriptionscited by 226
  12. 2021Keeping Your Eye on the Ball: Trajectory Attention in Video Transformerscited by 224
38 more
  1. 2021CLIP2Video: Mastering Video-Text Retrieval via Image CLIPcited by 223
  2. 2021VIOLET : End-to-End Video-Language Transformers with Masked Visual-token Modelingcited by 187
  3. 2023VAST: A Vision-Audio-Subtitle-Text Omni-Modality Foundation Model and Datasetcited by 168
  4. 2022Revisiting the "Video" in Video-Language Understandingcited by 167
  5. 2021SwinBERT: End-to-End Transformers with Sparse Attention for Video Captioningcited by 161
  6. 2021Align and Prompt: Video-and-Language Pre-training with Entity Promptscited by 149
  7. 2022OmniVL:One Foundation Model for Image-Language and Video-Language Taskscited by 143
  8. 2023VALOR: Vision-Audio-Language Omni-Perception Pretraining Model and Datasetcited by 138
  9. 2023EgoVLPv2: Egocentric Video-Language Pre-training with Fusion in the Backbonecited by 130
  10. 2022X-Pool: Cross-Modal Language-Video Attention for Text-Video Retrievalcited by 127
  11. 2022Language Models with Image Descriptors are Strong Few-Shot Video-Language Learnerscited by 117
  12. 2022End-to-end Generative Pretraining for Multimodal Video Captioningcited by 115
  13. 2022TS2-Net: Token Shift and Selection Transformer for Text-Video Retrievalcited by 104
  14. 2021VLM: Task-agnostic Video-Language Model Pre-training for Video Understandingcited by 102
  15. 2021VALUE: A Multi-Task Benchmark for Video-and-Language Understanding Evaluationcited by 95
  16. 2021Thinking Fast and Slow: Efficient Text-to-Visual Retrieval with Transformerscited by 93
  17. 2023Wider and Deeper LLM Networks are Fairer LLM Evaluatorscited by 91
  18. 2022Cap4Video: What Can Auxiliary Captions Do for Text-Video Retrieval?cited by 90
  19. 2021HiT: Hierarchical Transformer with Momentum Contrast for Video-Text Retrievalcited by 87
  20. 2022CenterCLIP: Token Clustering for Efficient Text-Video Retrievalcited by 86
  21. 2022Self-Supervised Learning for Videos: A Surveycited by 85
  22. 2021TEACHTEXT: CrossModal Generalized Distillation for Text-Video Retrievalcited by 81
  23. 2022Learning Audio-Video Modalities from Image Captionscited by 80
  24. 2021Everything at Once -- Multi-modal Fusion Transformer for Video Retrievalcited by 79
  25. 2022Bridging Video-text Retrieval with Multiple Choice Questionscited by 79
  26. 2023Knowing Where to Focus: Event-aware Transformer for Video Groundingcited by 79
  27. 2021MDMMT: Multidomain Multimodal Transformer for Video Retrievalcited by 78
  28. 2022Disentangled Representation Learning for Text-Video Retrievalcited by 77
  29. 2022An Empirical Study of End-to-End Video-Language Transformers with Masked Visual Modelingcited by 74
  30. 2022CLIP-ViP: Adapting Pre-trained Image-Text Model to Video-Language Representation Alignmentcited by 74
  31. 2022LAVENDER: Unifying Video-Language Understanding as Masked Language Modelingcited by 73
  32. 2021Cross Modal Retrieval with Querybank Normalisationcited by 71
  33. 2021A Straightforward Framework For Video Retrieval Using CLIPcited by 66
  34. 2022Video Transformers: A Surveycited by 63
  35. 2021Multimodal Clustering Networks for Self-supervised Learning from Unlabeled Videoscited by 62
  36. 2022Long-Form Video-Language Pre-Training with Multimodal Temporal Contrastive Learningcited by 59
  37. 2021Object-aware Video-language Pre-training for Retrievalcited by 57
  38. 2022A CLIP-Hitchhiker's Guide to Long Video Retrievalcited by 57

50 most cited of 170 All 170 (JSON)