# SHE-Net: Syntax-Hierarchy-Enhanced Text-Video Retrieval

arXiv: https://arxiv.org/abs/2404.14066 · HTML: https://mlsanity.com/paper/2404.14066/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 36 of its 55 references that are in the map.
- First posted on arXiv 22 Apr 2024.

## Builds on (most cited first, 30 of 36)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Learning to Prompt for Vision-Language Models](https://mlsanity.com/paper/2109.01134/) (2021)
- [Align before Fuse: Vision and Language Representation Learning with Momentum Distillation](https://mlsanity.com/paper/2107.07651/) (2021)
- [Frozen in Time: A Joint Video and Image Encoder for End-to-End Retrieval](https://mlsanity.com/paper/2104.00650/) (2021)
- [VL-BERT: Pre-training of Generic Visual-Linguistic Representations](https://mlsanity.com/paper/1908.08530/) (2019)
- [HowTo100M: Learning a Text-Video Embedding by Watching Hundred Million Narrated Video Clips](https://mlsanity.com/paper/1906.03327/) (2019)
- [Florence: A New Foundation Model for Computer Vision](https://mlsanity.com/paper/2111.11432/) (2021)
- [VideoBERT: A Joint Model for Video and Language Representation Learning](https://mlsanity.com/paper/1904.01766/) (2019)
- [Localizing Moments in Video with Natural Language](https://mlsanity.com/paper/1708.01641/) (2017)
- [CLIP4Clip: An Empirical Study of CLIP for End to End Video Clip Retrieval](https://mlsanity.com/paper/2104.08860/) (2021)
- [Unicoder-VL: A Universal Encoder for Vision and Language by Cross-modal Pre-training](https://mlsanity.com/paper/1908.06066/) (2019)
- [Multi-modal Transformer for Video Retrieval](https://mlsanity.com/paper/2007.10639/) (2020)
- [HERO: Hierarchical Encoder for Video+Language Omni-representation Pre-training](https://mlsanity.com/paper/2005.00200/) (2020)
- [ActionCLIP: A New Paradigm for Video Action Recognition](https://mlsanity.com/paper/2109.08472/) (2021)
- [X-CLIP: End-to-End Multi-grained Contrastive Learning for Video-Text Retrieval](https://mlsanity.com/paper/2207.07285/) (2022)
- [UniVL: A Unified Video and Language Pre-Training Model for Multimodal Understanding and Generation](https://mlsanity.com/paper/2002.06353/) (2020)
- [CLIP2Video: Mastering Video-Text Retrieval via Image CLIP](https://mlsanity.com/paper/2106.11097/) (2021)
- [Fine-grained Video-Text Retrieval with Hierarchical Graph Reasoning](https://mlsanity.com/paper/2003.00392/) (2020)
- [X-Pool: Cross-Modal Language-Video Attention for Text-Video Retrieval](https://mlsanity.com/paper/2203.15086/) (2022)
- [Improving Video-Text Retrieval by Multi-Stream Corpus Alignment and Dual Softmax Loss](https://mlsanity.com/paper/2109.04290/) (2021)
- [TS2-Net: Token Shift and Selection Transformer for Text-Video Retrieval](https://mlsanity.com/paper/2207.07852/) (2022)
- [Fine-Grained Action Retrieval Through Multiple Parts-of-Speech Embeddings](https://mlsanity.com/paper/1908.03477/) (2019)
- [Blindly Assess Quality of In-the-Wild Videos via Quality-aware Pre-training and Motion Perception](https://mlsanity.com/paper/2108.08505/) (2021)
- [CenterCLIP: Token Clustering for Efficient Text-Video Retrieval](https://mlsanity.com/paper/2205.00823/) (2022)
- [WenLan: Bridging Vision and Language by Large-Scale Multi-Modal Pre-Training](https://mlsanity.com/paper/2103.06561/) (2021)
- [Disentangled Representation Learning for Text-Video Retrieval](https://mlsanity.com/paper/2203.07111/) (2022)
- [Revisiting Temporal Modeling for CLIP-based Image-to-Video Knowledge Transferring](https://mlsanity.com/paper/2301.11116/) (2023)
- [Reading-strategy Inspired Visual Representation Learning for Text-to-Video Retrieval](https://mlsanity.com/paper/2201.09168/) (2022)
- [HANet: Hierarchical Alignment Networks for Video-Text Retrieval](https://mlsanity.com/paper/2107.12059/) (2021)

## Built on by (most cited first, 6 of 6)

- [Zero-shot Video Moment Retrieval via Off-the-shelf Multimodal Large Language Models](https://mlsanity.com/paper/2501.07972/) (2025)
- [Expertized Caption Auto-Enhancement for Video-Text Retrieval](https://mlsanity.com/paper/2502.02885/) (2025)
- [From Mapping to Composing: A Two-Stage Framework for Zero-shot Composed Image Retrieval](https://mlsanity.com/paper/2504.17990/) (2025)
- [Leveraging Auxiliary Information in Text-to-Video Retrieval: A Review](https://mlsanity.com/paper/2505.23952/) (2025)
- [Prompt-aware of Frame Sampling for Efficient Text-Video Retrieval](https://mlsanity.com/paper/2507.15491/) (2025)
- [EagleNet: Energy-Aware Fine-Grained Relationship Learning Network for Text-Video Retrieval](https://mlsanity.com/paper/2603.25267/) (2026)

## Abstract (arXiv)

The user base of short video apps has experienced unprecedented growth in recent years, resulting in a significant demand for video content analysis. In particular, text-video retrieval, which aims to find the top matching videos given text descriptions from a vast video corpus, is an essential function, the primary challenge of which is to bridge the modality gap. Nevertheless, most existing approaches treat texts merely as discrete tokens and neglect their syntax structures. Moreover, the abundant spatial and temporal clues in videos are often underutilized due to the lack of interaction with text. To address these issues, we argue that using texts as guidance to focus on relevant temporal frames and spatial regions within videos is beneficial. In this paper, we propose a novel Syntax-Hierarchy-Enhanced text-video retrieval method (SHE-Net) that exploits the inherent semantic and syntax hierarchy of texts to bridge the modality gap from two perspectives. First, to facilitate a more fine-grained integration of visual content, we employ the text syntax hierarchy, which reveals the grammatical structure of text descriptions, to guide the visual representations. Second, to further enhance the multi-modal interaction and alignment, we also utilize the syntax hierarchy to guide the similarity calculation. We evaluated our method on four public text-video retrieval datasets of MSR-VTT, MSVD, DiDeMo, and ActivityNet. The experimental results and ablation studies confirm the advantages of our proposed method.