# Text-Video Retrieval with Disentangled Conceptualization and Set-to-Set Alignment

arXiv: https://arxiv.org/abs/2305.12218 · HTML: https://mlsanity.com/paper/2305.12218/

## Key facts

- Cited by 32 papers in the map (data through 14 Oct 2026).
- Builds on 39 of its 63 references that are in the map.
- First posted on arXiv 20 May 2023.

## Builds on (most cited first, 30 of 39)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Representation Learning with Contrastive Predictive Coding](https://mlsanity.com/paper/1807.03748/) (2018)
- [Frozen in Time: A Joint Video and Image Encoder for End-to-End Retrieval](https://mlsanity.com/paper/2104.00650/) (2021)
- [InfoGAN: Interpretable Representation Learning by Information Maximizing Generative Adversarial Nets](https://mlsanity.com/paper/1606.03657/) (2016)
- [Dense-Captioning Events in Videos](https://mlsanity.com/paper/1705.00754/) (2017)
- [Localizing Moments in Video with Natural Language](https://mlsanity.com/paper/1708.01641/) (2017)
- [CLIP4Clip: An Empirical Study of CLIP for End to End Video Clip Retrieval](https://mlsanity.com/paper/2104.08860/) (2021)
- [Less is More: ClipBERT for Video-and-Language Learning via Sparse Sampling](https://mlsanity.com/paper/2102.06183/) (2021)
- [Multi-modal Transformer for Video Retrieval](https://mlsanity.com/paper/2007.10639/) (2020)
- [CLIP2Video: Mastering Video-Text Retrieval via Image CLIP](https://mlsanity.com/paper/2106.11097/) (2021)
- [Use What You Have: Video Retrieval Using Representations From Collaborative Experts](https://mlsanity.com/paper/1907.13487/) (2019)
- [A Dataset for Movie Description](https://mlsanity.com/paper/1501.02530/) (2015)
- [Fine-grained Video-Text Retrieval with Hierarchical Graph Reasoning](https://mlsanity.com/paper/2003.00392/) (2020)
- [Support-set bottlenecks for video-text representation learning](https://mlsanity.com/paper/2010.02824/) (2020)
- [X-Pool: Cross-Modal Language-Video Attention for Text-Video Retrieval](https://mlsanity.com/paper/2203.15086/) (2022)
- [Robustly Disentangled Causal Mechanisms: Validating Deep Representations for Interventional Robustness](https://mlsanity.com/paper/1811.00007/) (2018)
- [TS2-Net: Token Shift and Selection Transformer for Text-Video Retrieval](https://mlsanity.com/paper/2207.07852/) (2022)
- [T2VLAD: Global-Local Sequence Alignment for Text-Video Retrieval](https://mlsanity.com/paper/2104.10054/) (2021)
- [Learning Deep Disentangled Embeddings with the F-Statistic Loss](https://mlsanity.com/paper/1802.05312/) (2018)
- [Fine-Grained Action Retrieval Through Multiple Parts-of-Speech Embeddings](https://mlsanity.com/paper/1908.03477/) (2019)
- [Are Disentangled Representations Helpful for Abstract Visual Reasoning?](https://mlsanity.com/paper/1905.12506/) (2019)
- [TEACHTEXT: CrossModal Generalized Distillation for Text-Video Retrieval](https://mlsanity.com/paper/2104.08271/) (2021)
- [Disentangled Representation Learning for Text-Video Retrieval](https://mlsanity.com/paper/2203.07111/) (2022)
- [Cross Modal Retrieval with Querybank Normalisation](https://mlsanity.com/paper/2112.12777/) (2021)
- [Disentangling Factors of Variation Using Few Labels](https://mlsanity.com/paper/1905.01258/) (2019)
- [Theory and Evaluation Metrics for Learning Disentangled Representations](https://mlsanity.com/paper/1908.09961/) (2019)
- [A CLIP-Hitchhiker's Guide to Long Video Retrieval](https://mlsanity.com/paper/2205.08508/) (2022)

## Built on by (most cited first, 30 of 32)

- [Chat-UniVi: Unified Visual Representation Empowers Large Language Models with Image and Video Understanding](https://mlsanity.com/paper/2311.08046/) (2023)
- [Video Understanding with Large Language Models: A Survey](https://mlsanity.com/paper/2312.17432/) (2023)
- [MomentDiff: Generative Video Moment Retrieval from Random to Real](https://mlsanity.com/paper/2307.02869/) (2023)
- [Act As You Wish: Fine-Grained Control of Motion Diffusion Model with Hierarchical Semantic Graphs](https://mlsanity.com/paper/2311.01015/) (2023)
- [Text Is MASS: Modeling as Stochastic Embedding for Text-Video Retrieval](https://mlsanity.com/paper/2403.17998/) (2024)
- [FreestyleRet: Retrieving Images from Style-Diversified Queries](https://mlsanity.com/paper/2312.02428/) (2023)
- [HVD: Human Vision-Driven Video Representation Learning for Text-Video Retrieval](https://mlsanity.com/paper/2601.16155/) (2026)
- [GMMFormer v2: An Uncertainty-aware Framework for Partially Relevant Video Retrieval](https://mlsanity.com/paper/2405.13824/) (2024)
- [Ambiguity-Restrained Text-Video Representation Learning for Partially Relevant Video Retrieval](https://mlsanity.com/paper/2506.07471/) (2025)
- [MERLIN: Multimodal Embedding Refinement via LLM-based Iterative Navigation for Text-Video Retrieval-Rerank Pipeline](https://mlsanity.com/paper/2407.12508/) (2024)
- [Out-of-Distributed Semantic Pruning for Robust Semi-Supervised Learning](https://mlsanity.com/paper/2305.18158/) (2023)
- [MUSE: Mamba is Efficient Multi-scale Learner for Text-video Retrieval](https://mlsanity.com/paper/2408.10575/) (2024)
- [NeighborRetr: Balancing Hub Centrality in Cross-Modal Retrieval](https://mlsanity.com/paper/2503.10526/) (2025)
- [SEDS: Semantically Enhanced Dual-Stream Encoder for Sign Language Retrieval](https://mlsanity.com/paper/2407.16394/) (2024)
- [ICSVR: Investigating Compositional and Syntactic Understanding in Video Retrieval Models](https://mlsanity.com/paper/2306.16533/) (2023)
- [Improving Scene Graph Generation with Superpixel-Based Interaction Learning](https://mlsanity.com/paper/2308.02339/) (2023)
- [Hierarchical Banzhaf Interaction for General Video-Language Representation Learning](https://mlsanity.com/paper/2412.20964/) (2024)
- [Uneven Event Modeling for Partially Relevant Video Retrieval](https://mlsanity.com/paper/2506.00891/) (2025)
- [Text-Video Retrieval with Global-Local Semantic Consistent Learning](https://mlsanity.com/paper/2405.12710/) (2024)
- [Exploring Optimal Transport-Based Multi-Grained Alignments for Text-Molecule Retrieval](https://mlsanity.com/paper/2411.11875/) (2024)
- [Text Proxy: Decomposing Retrieval from a 1-to-N Relationship into N 1-to-1 Relationships for Text-Video Retrieval](https://mlsanity.com/paper/2410.06618/) (2024)
- [BiMa: Towards Biases Mitigation for Text-Video Retrieval via Scene Element Guidance](https://mlsanity.com/paper/2506.03589/) (2025)
- [Unifying Latent and Lexicon Representations for Effective Video-Text Retrieval](https://mlsanity.com/paper/2402.16769/) (2024)
- [Leveraging Modality Tags for Enhanced Cross-Modal Video Retrieval](https://mlsanity.com/paper/2504.01591/) (2025)
- [Rebalancing Contrastive Alignment with Bottlenecked Semantic Increments in Text-Video Retrieval](https://mlsanity.com/paper/2505.12499/) (2025)
- [MAVIS: Multi-Agent Video Retrieval via Structured Video Understanding](https://mlsanity.com/paper/2606.09641/) (2026)
- [Text-Video Retrieval via Variational Multi-Modal Hypergraph Networks](https://mlsanity.com/paper/2401.03177/) (2024)
- [DETACH : Decomposed Spatio-Temporal Alignment for Exocentric Video and Ambient Sensors with Staged Learning](https://mlsanity.com/paper/2512.20409/) (2025)
- [SAVE: Speech-Aware Video Representation Learning for Video-Text Retrieval](https://mlsanity.com/paper/2603.08224/) (2026)
- [Understanding the Performance Plateau in Text-to-Video Retrieval: A Comprehensive Empirical and Linguistic Analysis](https://mlsanity.com/paper/2605.00826/) (2026)

## Abstract (arXiv)

Text-video retrieval is a challenging cross-modal task, which aims to align visual entities with natural language descriptions. Current methods either fail to leverage the local details or are computationally expensive. What's worse, they fail to leverage the heterogeneous concepts in data. In this paper, we propose the Disentangled Conceptualization and Set-to-set Alignment (DiCoSA) to simulate the conceptualizing and reasoning process of human beings. For disentangled conceptualization, we divide the coarse feature into multiple latent factors related to semantic concepts. For set-to-set alignment, where a set of visual concepts correspond to a set of textual concepts, we propose an adaptive pooling method to aggregate semantic concepts to address the partial matching. In particular, since we encode concepts independently in only a few dimensions, DiCoSA is superior at efficiency and granularity, ensuring fine-grained interactions using a similar computational complexity as coarse-grained alignment. Extensive experiments on five datasets, including MSR-VTT, LSMDC, MSVD, ActivityNet, and DiDeMo, demonstrate that our method outperforms the existing state-of-the-art methods.