# Interpretable Embedding for Ad-hoc Video Search

arXiv: https://arxiv.org/abs/2402.11812 · HTML: https://mlsanity.com/paper/2402.11812/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 7 of its 49 references that are in the map.
- First posted on arXiv 19 Feb 2024.

## Builds on (most cited first, 7 of 7)

- [Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation](https://mlsanity.com/paper/1406.1078/) (2014)
- [Distributed Representations of Words and Phrases and their Compositionality](https://mlsanity.com/paper/1310.4546/) (2013)
- [Microsoft COCO Captions: Data Collection and Evaluation Server](https://mlsanity.com/paper/1504.00325/) (2015)
- [TGIF: A New Dataset and Benchmark on Animated GIF Description](https://mlsanity.com/paper/1604.02748/) (2016)
- [Dual Encoding for Zero-Example Video Retrieval](https://mlsanity.com/paper/1809.06181/) (2018)
- [Predicting Visual Features from Text for Image and Video Caption Retrieval](https://mlsanity.com/paper/1709.01362/) (2017)
- [TRECVID 2019: An Evaluation Campaign to Benchmark Video Activity Detection, Video Captioning and Matching, and Video Search & Retrieval](https://mlsanity.com/paper/2009.09984/) (2020)

## Built on by (most cited first, 4 of 4)

- [Improving Interpretable Embeddings for Ad-hoc Video Search with Generative Captions and Multi-word Concept Bank](https://mlsanity.com/paper/2404.06173/) (2024)
- [Multimodal LLM-based Query Paraphrasing for Video Search](https://mlsanity.com/paper/2407.12341/) (2024)
- [Learning Partially-Decorrelated Common Spaces for Ad-hoc Video Search](https://mlsanity.com/paper/2508.02340/) (2025)
- [Adaptive Multi-Agent Reasoning for Text-to-Video Retrieval](https://mlsanity.com/paper/2602.19040/) (2025)

## Abstract (arXiv)

Answering query with semantic concepts has long been the mainstream approach for video search. Until recently, its performance is surpassed by concept-free approach, which embeds queries in a joint space as videos. Nevertheless, the embedded features as well as search results are not interpretable, hindering subsequent steps in video browsing and query reformulation. This paper integrates feature embedding and concept interpretation into a neural network for unified dual-task learning. In this way, an embedding is associated with a list of semantic concepts as an interpretation of video content. This paper empirically demonstrates that, by using either the embedding features or concepts, considerable search improvement is attainable on TRECVid benchmarked datasets. Concepts are not only effective in pruning false positive videos, but also highly complementary to concept-free search, leading to large margin of improvement compared to state-of-the-art approaches.