# SEA: Sentence Encoder Assembly for Video Retrieval by Textual Queries

arXiv: https://arxiv.org/abs/2011.12091 · HTML: https://mlsanity.com/paper/2011.12091/

## Key facts

- Cited by 26 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 24 Nov 2020.

## Built on by (most cited first, 26 of 26)

- [Video Corpus Moment Retrieval with Contrastive Learning](https://mlsanity.com/paper/2105.06247/) (2021)
- [Temporal Sentence Grounding in Videos: A Survey and Future Directions](https://mlsanity.com/paper/2201.08071/) (2022)
- [Reading-strategy Inspired Visual Representation Learning for Text-to-Video Retrieval](https://mlsanity.com/paper/2201.09168/) (2022)
- [HANet: Hierarchical Alignment Networks for Video-Text Retrieval](https://mlsanity.com/paper/2107.12059/) (2021)
- [Lightweight Attentional Feature Fusion: A New Baseline for Text-to-Video Retrieval](https://mlsanity.com/paper/2112.01832/) (2021)
- [Cross-Lingual Cross-Modal Retrieval with Noise-Robust Learning](https://mlsanity.com/paper/2208.12526/) (2022)
- [Learn to Understand Negation in Video Retrieval](https://mlsanity.com/paper/2205.00132/) (2022)
- [A Feature-space Multimodal Data Augmentation Technique for Text-video Retrieval](https://mlsanity.com/paper/2208.02080/) (2022)
- [Towards Efficient and Effective Text-to-Video Retrieval with Coarse-to-Fine Visual Representation Learning](https://mlsanity.com/paper/2401.00701/) (2024)
- [MDMMT-2: Multidomain Multimodal Transformer for Video Retrieval, One More Step Towards Generalization](https://mlsanity.com/paper/2203.07086/) (2022)
- [BiC-Net: Learning Efficient Spatio-Temporal Relation for Text-Video Retrieval](https://mlsanity.com/paper/2110.15609/) (2021)
- [Boosting Video-Text Retrieval with Explicit High-Level Semantics](https://mlsanity.com/paper/2208.04215/) (2022)
- [PRVR: Partially Relevant Video Retrieval](https://mlsanity.com/paper/2208.12510/) (2022)
- [Hybrid-Tower: Fine-grained Pseudo-query Interaction and Generation for Text-to-Video Retrieval](https://mlsanity.com/paper/2509.04773/) (2025)
- [(Un)likelihood Training for Interpretable Embedding](https://mlsanity.com/paper/2207.00282/) (2022)
- [Improving Interpretable Embeddings for Ad-hoc Video Search with Generative Captions and Multi-word Concept Bank](https://mlsanity.com/paper/2404.06173/) (2024)
- [Dual Learning with Dynamic Knowledge Distillation and Soft Alignment for Partially Relevant Video Retrieval](https://mlsanity.com/paper/2510.12283/) (2025)
- [Efficient Cross-Modal Video Retrieval with Meta-Optimized Frames](https://mlsanity.com/paper/2210.08452/) (2022)
- [Renmin University of China at TRECVID 2022: Improving Video Search by Feature Fusion and Negation Understanding](https://mlsanity.com/paper/2211.15039/) (2022)
- [TeachCLIP: Multi-Grained Teaching for Efficient Text-to-Video Retrieval](https://mlsanity.com/paper/2308.01217/) (2023)
- [Multimodal LLM-based Query Paraphrasing for Video Search](https://mlsanity.com/paper/2407.12341/) (2024)
- [Leveraging Auxiliary Information in Text-to-Video Retrieval: A Review](https://mlsanity.com/paper/2505.23952/) (2025)
- [Are All Combinations Equal? Combining Textual and Visual Features with Multiple Space Learning for Text-Based Video Retrieval](https://mlsanity.com/paper/2211.11351/) (2022)
- [Learning Partially-Decorrelated Common Spaces for Ad-hoc Video Search](https://mlsanity.com/paper/2508.02340/) (2025)
- [Adaptive Multi-Agent Reasoning for Text-to-Video Retrieval](https://mlsanity.com/paper/2602.19040/) (2025)
- [Understanding the Performance Plateau in Text-to-Video Retrieval: A Comprehensive Empirical and Linguistic Analysis](https://mlsanity.com/paper/2605.00826/) (2026)

## Abstract (arXiv)

Retrieving unlabeled videos by textual queries, known as Ad-hoc Video Search (AVS), is a core theme in multimedia data management and retrieval. The success of AVS counts on cross-modal representation learning that encodes both query sentences and videos into common spaces for semantic similarity computation. Inspired by the initial success of previously few works in combining multiple sentence encoders, this paper takes a step forward by developing a new and general method for effectively exploiting diverse sentence encoders. The novelty of the proposed method, which we term Sentence Encoder Assembly (SEA), is two-fold. First, different from prior art that use only a single common space, SEA supports text-video matching in multiple encoder-specific common spaces. Such a property prevents the matching from being dominated by a specific encoder that produces an encoding vector much longer than other encoders. Second, in order to explore complementarities among the individual common spaces, we propose multi-space multi-loss learning. As extensive experiments on four benchmarks (MSR-VTT, TRECVID AVS 2016-2019, TGIF and MSVD) show, SEA surpasses the state-of-the-art. In addition, SEA is extremely ease to implement. All this makes SEA an appealing solution for AVS and promising for continuously advancing the task by harvesting new sentence encoders.