# Tree-Augmented Cross-Modal Encoding for Complex-Query Video Retrieval

arXiv: https://arxiv.org/abs/2007.02503 · HTML: https://mlsanity.com/paper/2007.02503/

## Key facts

- Cited by 47 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 6 Jul 2020.

## Built on by (most cited first, 30 of 47)

- [X-CLIP: End-to-End Multi-grained Contrastive Learning for Video-Text Retrieval](https://mlsanity.com/paper/2207.07285/) (2022)
- [Interventional Video Grounding with Dual Contrastive Learning](https://mlsanity.com/paper/2106.11013/) (2021)
- [Deconfounded Video Moment Retrieval with Causal Intervention](https://mlsanity.com/paper/2106.01534/) (2021)
- [Adaptive Proposal Generation Network for Temporal Sentence Localization in Videos](https://mlsanity.com/paper/2109.06398/) (2021)
- [Finding and Editing Multi-Modal Neurons in Pre-Trained Transformers](https://mlsanity.com/paper/2311.07470/) (2023)
- [Cross-Silo Prototypical Calibration for Federated Learning with Non-IID Data](https://mlsanity.com/paper/2308.03457/) (2023)
- [Reading-strategy Inspired Visual Representation Learning for Text-to-Video Retrieval](https://mlsanity.com/paper/2201.09168/) (2022)
- [HANet: Hierarchical Alignment Networks for Video-Text Retrieval](https://mlsanity.com/paper/2107.12059/) (2021)
- [A Review of Deep Learning for Video Captioning](https://mlsanity.com/paper/2304.11431/) (2023)
- [Lightweight Attentional Feature Fusion: A New Baseline for Text-to-Video Retrieval](https://mlsanity.com/paper/2112.01832/) (2021)
- [Classification-Then-Grounding: Reformulating Video Scene Graphs as Temporal Bipartite Graphs](https://mlsanity.com/paper/2112.04222/) (2021)
- [DFIL: Deepfake Incremental Learning by Exploiting Domain-invariant Forgery Clues](https://mlsanity.com/paper/2309.09526/) (2023)
- [Delving Deeper: Hierarchical Visual Perception for Robust Video-Text Retrieval](https://mlsanity.com/paper/2601.12768/) (2026)
- [Cross-Lingual Cross-Modal Retrieval with Noise-Robust Learning](https://mlsanity.com/paper/2208.12526/) (2022)
- [Unified Multi-modal Unsupervised Representation Learning for Skeleton-based Action Understanding](https://mlsanity.com/paper/2311.03106/) (2023)
- [Rethinking the Video Sampling and Reasoning Strategies for Temporal Sentence Grounding](https://mlsanity.com/paper/2301.00514/) (2023)
- [Progressive Localization Networks for Language-based Moment Localization](https://mlsanity.com/paper/2102.01282/) (2021)
- [Neural ranking models for document retrieval](https://mlsanity.com/paper/2102.11903/) (2021)
- [Learn to Understand Negation in Video Retrieval](https://mlsanity.com/paper/2205.00132/) (2022)
- [Deep Learning for Video-Text Retrieval: a Review](https://mlsanity.com/paper/2302.12552/) (2023)
- [Fine-Grained Fashion Similarity Prediction by Attribute-Specific Embedding Learning](https://mlsanity.com/paper/2104.02429/) (2021)
- [Semantic Data Augmentation based Distance Metric Learning for Domain Generalization](https://mlsanity.com/paper/2208.02803/) (2022)
- [CL2CM: Improving Cross-Lingual Cross-Modal Retrieval via Cross-Lingual Knowledge Transfer](https://mlsanity.com/paper/2312.08984/) (2023)
- [From Region to Patch: Attribute-Aware Foreground-Background Contrastive Learning for Fine-Grained Fashion Retrieval](https://mlsanity.com/paper/2305.10260/) (2023)
- [Class-level Structural Relation Modelling and Smoothing for Visual Representation Learning](https://mlsanity.com/paper/2308.04142/) (2023)
- [BiC-Net: Learning Efficient Spatio-Temporal Relation for Text-Video Retrieval](https://mlsanity.com/paper/2110.15609/) (2021)
- [Hybrid Contrastive Quantization for Efficient Cross-View Video Retrieval](https://mlsanity.com/paper/2202.03384/) (2022)
- [CMMLoc: Advancing Text-to-PointCloud Localization with Cauchy-Mixture-Model Based Framework](https://mlsanity.com/paper/2503.02593/) (2025)
- [BP-Triplet Net for Unsupervised Domain Adaptation: A Bayesian Perspective](https://mlsanity.com/paper/2202.09541/) (2022)
- [PRVR: Partially Relevant Video Retrieval](https://mlsanity.com/paper/2208.12510/) (2022)

## Abstract (arXiv)

The rapid growth of user-generated videos on the Internet has intensified the need for text-based video retrieval systems. Traditional methods mainly favor the concept-based paradigm on retrieval with simple queries, which are usually ineffective for complex queries that carry far more complex semantics. Recently, embedding-based paradigm has emerged as a popular approach. It aims to map the queries and videos into a shared embedding space where semantically-similar texts and videos are much closer to each other. Despite its simplicity, it forgoes the exploitation of the syntactic structure of text queries, making it suboptimal to model the complex queries. To facilitate video retrieval with complex queries, we propose a Tree-augmented Cross-modal Encoding method by jointly learning the linguistic structure of queries and the temporal representation of videos. Specifically, given a complex user query, we first recursively compose a latent semantic tree to structurally describe the text query. We then design a tree-augmented query encoder to derive structure-aware query representation and a temporal attentive video encoder to model the temporal characteristics of videos. Finally, both the query and videos are mapped into a joint embedding space for matching and ranking. In this approach, we have a better understanding and modeling of the complex queries, thereby achieving a better video retrieval performance. Extensive experiments on large scale video retrieval benchmark datasets demonstrate the effectiveness of our approach.