# Video-Text Retrieval by Supervised Sparse Multi-Grained Learning

arXiv: https://arxiv.org/abs/2302.09473 · HTML: https://mlsanity.com/paper/2302.09473/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 41 of its 66 references that are in the map.
- First posted on arXiv 19 Feb 2023.

## Builds on (most cited first, 30 of 41)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [SGDR: Stochastic Gradient Descent with Warm Restarts](https://mlsanity.com/paper/1608.03983/) (2016)
- [Dense Passage Retrieval for Open-Domain Question Answering](https://mlsanity.com/paper/2004.04906/) (2020)
- [Is Space-Time Attention All You Need for Video Understanding?](https://mlsanity.com/paper/2102.05095/) (2021)
- [Oscar: Object-Semantics Aligned Pre-training for Vision-Language Tasks](https://mlsanity.com/paper/2004.06165/) (2020)
- [Frozen in Time: A Joint Video and Image Encoder for End-to-End Retrieval](https://mlsanity.com/paper/2104.00650/) (2021)
- [HowTo100M: Learning a Text-Video Embedding by Watching Hundred Million Narrated Video Clips](https://mlsanity.com/paper/1906.03327/) (2019)
- [FLAVA: A Foundational Language And Vision Alignment Model](https://mlsanity.com/paper/2112.04482/) (2021)
- [VideoCLIP: Contrastive Pre-training for Zero-shot Video-Text Understanding](https://mlsanity.com/paper/2109.14084/) (2021)
- [Mind the Gap: Understanding the Modality Gap in Multi-modal Contrastive Representation Learning](https://mlsanity.com/paper/2203.02053/) (2022)
- [End-to-End Learning of Visual Representations from Uncurated Instructional Videos](https://mlsanity.com/paper/1912.06430/) (2019)
- [Less is More: ClipBERT for Video-and-Language Learning via Sparse Sampling](https://mlsanity.com/paper/2102.06183/) (2021)
- [Multi-modal Transformer for Video Retrieval](https://mlsanity.com/paper/2007.10639/) (2020)
- [HERO: Hierarchical Encoder for Video+Language Omni-representation Pre-training](https://mlsanity.com/paper/2005.00200/) (2020)
- [Large-Scale Adversarial Training for Vision-and-Language Representation Learning](https://mlsanity.com/paper/2006.06195/) (2020)
- [X-CLIP: End-to-End Multi-grained Contrastive Learning for Video-Text Retrieval](https://mlsanity.com/paper/2207.07285/) (2022)
- [ActBERT: Learning Global-Local Video-Text Representations](https://mlsanity.com/paper/2011.07231/) (2020)
- [A Joint Sequence Fusion Model for Video Question Answering and Retrieval](https://mlsanity.com/paper/1808.02559/) (2018)
- [X-Pool: Cross-Modal Language-Video Attention for Text-Video Retrieval](https://mlsanity.com/paper/2203.15086/) (2022)
- [Improving Video-Text Retrieval by Multi-Stream Corpus Alignment and Dual Softmax Loss](https://mlsanity.com/paper/2109.04290/) (2021)
- [COIL: Revisit Exact Lexical Match in Information Retrieval with Contextualized Inverted List](https://mlsanity.com/paper/2104.07186/) (2021)
- [TS2-Net: Token Shift and Selection Transformer for Text-Video Retrieval](https://mlsanity.com/paper/2207.07852/) (2022)
- [VLM: Task-agnostic Video-Language Model Pre-training for Video Understanding](https://mlsanity.com/paper/2105.09996/) (2021)
- [CenterCLIP: Token Clustering for Efficient Text-Video Retrieval](https://mlsanity.com/paper/2205.00823/) (2022)
- [Multimodal Federated Learning via Contrastive Representation Ensemble](https://mlsanity.com/paper/2302.08888/) (2023)
- [Dual Encoding for Zero-Example Video Retrieval](https://mlsanity.com/paper/1809.06181/) (2018)
- [TEACHTEXT: CrossModal Generalized Distillation for Text-Video Retrieval](https://mlsanity.com/paper/2104.08271/) (2021)

## Built on by (most cited first, 7 of 7)

- [MultiVENT 2.0: A Massive Multilingual Benchmark for Event-Centric Video Retrieval](https://mlsanity.com/paper/2410.11619/) (2024)
- [To Align or Not to Align: Strategic Multimodal Representation Alignment for Optimal Performance](https://mlsanity.com/paper/2511.12121/) (2025)
- [Fewer Steps, Better Performance: Efficient Cross-Modal Clip Trimming for Video Moment Retrieval Using Language](https://mlsanity.com/paper/2605.29793/) (2026)
- [LEO-MINI: An Efficient Multimodal Large Language Model using Conditional Token Reduction and Mixture of Multi-Modal Experts](https://mlsanity.com/paper/2504.04653/) (2025)
- [DREAM: Improving Video-Text Retrieval Through Relevance-Based Augmentation Using Large Foundation Models](https://mlsanity.com/paper/2404.05083/) (2024)
- [RANKVIDEO: Reasoning Reranking for Text-to-Video Retrieval](https://mlsanity.com/paper/2602.02444/) (2026)
- [Turing Patterns for Multimedia: Reaction-Diffusion Multi-Modal Fusion for Language-Guided Video Moment Retrieval](https://mlsanity.com/paper/2606.01615/) (2026)

## Abstract (arXiv)

While recent progress in video-text retrieval has been advanced by the exploration of better representation learning, in this paper, we present a novel multi-grained sparse learning framework, S3MA, to learn an aligned sparse space shared between the video and the text for video-text retrieval. The shared sparse space is initialized with a finite number of sparse concepts, each of which refers to a number of words. With the text data at hand, we learn and update the shared sparse space in a supervised manner using the proposed similarity and alignment losses. Moreover, to enable multi-grained alignment, we incorporate frame representations for better modeling the video modality and calculating fine-grained and coarse-grained similarities. Benefiting from the learned shared sparse space and multi-grained similarities, extensive experiments on several video-text retrieval benchmarks demonstrate the superiority of S3MA over existing methods. Our code is available at https://github.com/yimuwangcs/Better_Cross_Modal_Retrieval.