# Text-Adaptive Multiple Visual Prototype Matching for Video-Text Retrieval

arXiv: https://arxiv.org/abs/2209.13307 · HTML: https://mlsanity.com/paper/2209.13307/

## Key facts

- Cited by 20 papers in the map (data through 14 Oct 2026).
- Builds on 31 of its 42 references that are in the map.
- First posted on arXiv 27 Sep 2022.

## Builds on (most cited first, 30 of 31)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation](https://mlsanity.com/paper/2201.12086/) (2022)
- [DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter](https://mlsanity.com/paper/1910.01108/) (2019)
- [ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Tasks](https://mlsanity.com/paper/1908.02265/) (2019)
- [Is Space-Time Attention All You Need for Video Understanding?](https://mlsanity.com/paper/2102.05095/) (2021)
- [Frozen in Time: A Joint Video and Image Encoder for End-to-End Retrieval](https://mlsanity.com/paper/2104.00650/) (2021)
- [VICReg: Variance-Invariance-Covariance Regularization for Self-Supervised Learning](https://mlsanity.com/paper/2105.04906/) (2021)
- [HowTo100M: Learning a Text-Video Embedding by Watching Hundred Million Narrated Video Clips](https://mlsanity.com/paper/1906.03327/) (2019)
- [Localizing Moments in Video with Natural Language](https://mlsanity.com/paper/1708.01641/) (2017)
- [Less is More: ClipBERT for Video-and-Language Learning via Sparse Sampling](https://mlsanity.com/paper/2102.06183/) (2021)
- [Multi-modal Transformer for Video Retrieval](https://mlsanity.com/paper/2007.10639/) (2020)
- [ActBERT: Learning Global-Local Video-Text Representations](https://mlsanity.com/paper/2011.07231/) (2020)
- [Use What You Have: Video Retrieval Using Representations From Collaborative Experts](https://mlsanity.com/paper/1907.13487/) (2019)
- [A Joint Sequence Fusion Model for Video Question Answering and Retrieval](https://mlsanity.com/paper/1808.02559/) (2018)
- [Unifying Visual-Semantic Embeddings with Multimodal Neural Language Models](https://mlsanity.com/paper/1411.2539/) (2014)
- [A Dataset for Movie Description](https://mlsanity.com/paper/1501.02530/) (2015)
- [Movie Description](https://mlsanity.com/paper/1605.03705/) (2016)
- [Fine-grained Video-Text Retrieval with Hierarchical Graph Reasoning](https://mlsanity.com/paper/2003.00392/) (2020)
- [Support-set bottlenecks for video-text representation learning](https://mlsanity.com/paper/2010.02824/) (2020)
- [Probabilistic Embeddings for Cross-Modal Retrieval](https://mlsanity.com/paper/2101.05068/) (2021)
- [AVLnet: Learning Audio-Visual Language Representations from Instructional Videos](https://mlsanity.com/paper/2006.09199/) (2020)
- [Learning a Text-Video Embedding from Incomplete and Heterogeneous Data](https://mlsanity.com/paper/1804.02516/) (2018)
- [Cross-Modal and Hierarchical Modeling of Video and Text](https://mlsanity.com/paper/1810.07212/) (2018)
- [TACo: Token-aware Cascade Contrastive Learning for Video-Text Alignment](https://mlsanity.com/paper/2108.09980/) (2021)
- [Translating Videos to Natural Language Using Deep Recurrent Neural Networks](https://mlsanity.com/paper/1412.4729/) (2014)
- [HiT: Hierarchical Transformer with Momentum Contrast for Video-Text Retrieval](https://mlsanity.com/paper/2103.15049/) (2021)
- [TEACHTEXT: CrossModal Generalized Distillation for Text-Video Retrieval](https://mlsanity.com/paper/2104.08271/) (2021)
- [Noise Estimation Using Density Estimation for Self-Supervised Multimodal Learning](https://mlsanity.com/paper/2003.03186/) (2020)
- [On Semantic Similarity in Video Retrieval](https://mlsanity.com/paper/2103.10095/) (2021)

## Built on by (most cited first, 20 of 20)

- [Cap4Video: What Can Auxiliary Captions Do for Text-Video Retrieval?](https://mlsanity.com/paper/2301.00184/) (2022)
- [UATVR: Uncertainty-Adaptive Text-Video Retrieval](https://mlsanity.com/paper/2301.06309/) (2023)
- [Text Is MASS: Modeling as Stochastic Embedding for Text-Video Retrieval](https://mlsanity.com/paper/2403.17998/) (2024)
- [Ambiguity-Restrained Text-Video Representation Learning for Partially Relevant Video Retrieval](https://mlsanity.com/paper/2506.07471/) (2025)
- [Prototypes are Balanced Units for Efficient and Effective Partially Relevant Video Retrieval](https://mlsanity.com/paper/2504.13035/) (2025)
- [Learning Trajectory-Word Alignments for Video-Language Tasks](https://mlsanity.com/paper/2301.01953/) (2023)
- [SEDS: Semantically Enhanced Dual-Stream Encoder for Sign Language Retrieval](https://mlsanity.com/paper/2407.16394/) (2024)
- [Bridging Information Asymmetry in Text-video Retrieval: A Data-centric Approach](https://mlsanity.com/paper/2408.07249/) (2024)
- [Quantifying and Narrowing the Unknown: Interactive Text-to-Video Retrieval via Uncertainty Minimization](https://mlsanity.com/paper/2507.15504/) (2025)
- [WAVER: Writing-style Agnostic Text-Video Retrieval via Distilling Vision-Language Models Through Open-Vocabulary Knowledge](https://mlsanity.com/paper/2312.09507/) (2023)
- [Visual-Semantic Decomposition and Partial Alignment for Document-based Zero-Shot Learning](https://mlsanity.com/paper/2407.15613/) (2024)
- [FedSC: Federated Learning with Semantic-Aware Collaboration](https://mlsanity.com/paper/2506.21012/) (2025)
- [Text Proxy: Decomposing Retrieval from a 1-to-N Relationship into N 1-to-1 Relationships for Text-Video Retrieval](https://mlsanity.com/paper/2410.06618/) (2024)
- [BiMa: Towards Biases Mitigation for Text-Video Retrieval via Scene Element Guidance](https://mlsanity.com/paper/2506.03589/) (2025)
- [Video-Language Alignment via Spatio-Temporal Graph Transformer](https://mlsanity.com/paper/2407.11677/) (2024)
- [Prompt-aware of Frame Sampling for Efficient Text-Video Retrieval](https://mlsanity.com/paper/2507.15491/) (2025)
- [T2VParser: Adaptive Decomposition Tokens for Partial Alignment in Text to Video Retrieval](https://mlsanity.com/paper/2507.20518/) (2025)
- [Learning Partially-Decorrelated Common Spaces for Ad-hoc Video Search](https://mlsanity.com/paper/2508.02340/) (2025)
- [Repeating Words for Video-Language Retrieval with Coarse-to-Fine Objectives](https://mlsanity.com/paper/2508.14812/) (2025)
- [EagleNet: Energy-Aware Fine-Grained Relationship Learning Network for Text-Video Retrieval](https://mlsanity.com/paper/2603.25267/) (2026)

## Abstract (arXiv)

Cross-modal retrieval between videos and texts has gained increasing research interest due to the rapid emergence of videos on the web. Generally, a video contains rich instance and event information and the query text only describes a part of the information. Thus, a video can correspond to multiple different text descriptions and queries. We call this phenomenon the ``Video-Text Correspondence Ambiguity'' problem. Current techniques mostly concentrate on mining local or multi-level alignment between contents of a video and text (textite.g., object to entity and action to verb). It is difficult for these methods to alleviate the video-text correspondence ambiguity by describing a video using only one single feature, which is required to be matched with multiple different text features at the same time. To address this problem, we propose a Text-Adaptive Multiple Visual Prototype Matching model, which automatically captures multiple prototypes to describe a video by adaptive aggregation of video token features. Given a query text, the similarity is determined by the most similar prototype to find correspondence in the video, which is termed text-adaptive matching. To learn diverse prototypes for representing the rich information in videos, we propose a variance loss to encourage different prototypes to attend to different contents of the video. Our method outperforms state-of-the-art methods on four public video retrieval datasets.