mlsanityData to 14 Oct 2026

Normalized Contrastive Learning for Text-Video Retrieval

arXiv 2212.11790 ↗First posted 30 Nov 2022cs.IR cs.CV cs.LGcited by 8

Not on the map. Why?

Abstract (arXiv)

Cross-modal contrastive learning has led the recent advances in multimodal retrieval with its simplicity and effectiveness. In this work, however, we reveal that cross-modal contrastive learning suffers from incorrect normalization of the sum retrieval probabilities of each text or video instance. Specifically, we show that many test instances are either over- or under-represented during retrieval, significantly hurting the retrieval performance. To address this problem, we propose Normalized Contrastive Learning (NCL) which utilizes the Sinkhorn-Knopp algorithm to compute the instance-wise biases that properly normalize the sum retrieval probabilities of each instance so that every text and video instance is fairly represented during cross-modal retrieval. Empirical study shows that NCL brings consistent and significant gains in text-video retrieval on different model architectures, with new state-of-the-art multimodal retrieval metrics on the ActivityNet, MSVD, and MSR-VTT datasets without any architecture engineering.

Key facts. Cited by 8 papers in the map (data through 14 Oct 2026). Builds on 35 of its 48 references that are in the map. First posted on arXiv 30 Nov 2022.

Builds on

35 of 48 references.

References · 35

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  5. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 12,049
  6. 2018Representation Learning with Contrastive Predictive Codingcited by 7,777
  7. 2016SGDR: Stochastic Gradient Descent with Warm Restartscited by 5,045
  8. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,322
  9. 2020Unsupervised Learning of Visual Features by Contrasting Cluster Assignmentscited by 2,857
  10. 2013Sinkhorn Distances: Lightspeed Computation of Optimal Transportation Distancescited by 2,316
  11. 2018Unsupervised Feature Learning via Non-Parametric Instance-level Discriminationcited by 1,845
  12. 2021Align before Fuse: Vision and Language Representation Learning with Momentum Distillationcited by 1,619
23 more
  1. 2020Understanding Contrastive Representation Learning through Alignment and Uniformity on the Hyperspherecited by 1,405
  2. 2019Contrastive Multiview Codingcited by 1,301
  3. 2021Frozen in Time: A Joint Video and Image Encoder for End-to-End Retrievalcited by 1,177
  4. 2017A Closer Look at Spatiotemporal Convolutions for Action Recognitioncited by 1,143
  5. 2019HowTo100M: Learning a Text-Video Embedding by Watching Hundred Million Narrated Video Clipscited by 950
  6. 2017Dense-Captioning Events in Videoscited by 793
  7. 2021CLIP4Clip: An Empirical Study of CLIP for End to End Video Clip Retrievalcited by 651
  8. 2020Contrastive Learning of Medical Visual Representations from Paired Images and Textcited by 641
  9. 2021VideoCLIP: Contrastive Pre-training for Zero-shot Video-Text Understandingcited by 602
  10. 2019Self-labelling via simultaneous clustering and representation learningcited by 528
  11. 2019End-to-End Learning of Visual Representations from Uncurated Instructional Videoscited by 507
  12. 2020Multi-modal Transformer for Video Retrievalcited by 395
  13. 2019Use What You Have: Video Retrieval Using Representations From Collaborative Expertscited by 214
  14. 2018A Joint Sequence Fusion Model for Video Question Answering and Retrievalcited by 208
  15. 2014Unifying Visual-Semantic Embeddings with Multimodal Neural Language Modelscited by 198
  16. 2020Support-set bottlenecks for video-text representation learningcited by 170
  17. 2019Large-scale weakly-supervised pre-training for video action recognitioncited by 130
  18. 2018Cross-Modal and Hierarchical Modeling of Video and Textcited by 103
  19. 2019Fine-Grained Action Retrieval Through Multiple Parts-of-Speech Embeddingscited by 101
  20. 2020Labelling unlabelled videos from scratch with multi-modal self-supervisioncited by 100
  21. 2021HiT: Hierarchical Transformer with Momentum Contrast for Video-Text Retrievalcited by 87
  22. 2021TEACHTEXT: CrossModal Generalized Distillation for Text-Video Retrievalcited by 81
  23. 2020Noise Estimation Using Density Estimation for Self-Supervised Multimodal Learningcited by 74

Built on by

Cited by year

Citing papers per year: 2023: 5; 2024: 2; 2025: 1.

8 papers.

  1. 2023Unified Coarse-to-Fine Alignment for Video-Text Retrievalcited by 49
  2. 2023Learning Visual Representations via Language-Guided Samplingcited by 32
  3. 2023Learning to Ground Instructional Articles in Videos through Narrationscited by 28
  4. 2023Balance Act: Mitigating Hubness in Cross-Modal Retrieval with Query and Gallery Bankscited by 20
  5. 2023Video-Text Retrieval by Supervised Sparse Multi-Grained Learningcited by 7
  6. 2024TokenBinder: Text-Video Retrieval with One-to-Many Alignment Paradigmcited by 6
  7. 2024DREAM: Improving Video-Text Retrieval Through Relevance-Based Augmentation Using Large Foundation Modelscited by 3
  8. 2025Impact of Language Guidance: A Reproducibility Studycited by 0