# Learn to Understand Negation in Video Retrieval

arXiv: https://arxiv.org/abs/2205.00132 · HTML: https://mlsanity.com/paper/2205.00132/

## Key facts

- Cited by 16 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 39 references that are in the map.
- First posted on arXiv 30 Apr 2022.

## Builds on (most cited first, 22 of 22)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Deep Visual-Semantic Alignments for Generating Image Descriptions](https://mlsanity.com/paper/1412.2306/) (2014)
- [CLIP4Clip: An Empirical Study of CLIP for End to End Video Clip Retrieval](https://mlsanity.com/paper/2104.08860/) (2021)
- [VATEX: A Large-Scale, High-Quality Multilingual Dataset for Video-and-Language Research](https://mlsanity.com/paper/1904.03493/) (2019)
- [Multi-modal Transformer for Video Retrieval](https://mlsanity.com/paper/2007.10639/) (2020)
- [What BERT is not: Lessons from a new suite of psycholinguistic diagnostics for language models](https://mlsanity.com/paper/1907.13528/) (2019)
- [Negated and Misprimed Probes for Pretrained Language Models: Birds Can Talk, But Cannot Fly](https://mlsanity.com/paper/1911.03343/) (2019)
- [CLIP2Video: Mastering Video-Text Retrieval via Image CLIP](https://mlsanity.com/paper/2106.11097/) (2021)
- [Use What You Have: Video Retrieval Using Representations From Collaborative Experts](https://mlsanity.com/paper/1907.13487/) (2019)
- [A Joint Sequence Fusion Model for Video Question Answering and Retrieval](https://mlsanity.com/paper/1808.02559/) (2018)
- [Fine-grained Video-Text Retrieval with Hierarchical Graph Reasoning](https://mlsanity.com/paper/2003.00392/) (2020)
- [MDMMT: Multidomain Multimodal Transformer for Video Retrieval](https://mlsanity.com/paper/2103.10699/) (2021)
- [Understanding by Understanding Not: Modeling Negation in Language Models](https://mlsanity.com/paper/2105.03519/) (2021)
- [Tree-Augmented Cross-Modal Encoding for Complex-Query Video Retrieval](https://mlsanity.com/paper/2007.02503/) (2020)
- [NegBERT: A Transfer Learning Approach for Negation Detection and Scope Resolution](https://mlsanity.com/paper/1911.04211/) (2019)
- [HANet: Hierarchical Alignment Networks for Video-Text Retrieval](https://mlsanity.com/paper/2107.12059/) (2021)
- [SEA: Sentence Encoder Assembly for Video Retrieval by Textual Queries](https://mlsanity.com/paper/2011.12091/) (2020)
- [Lightweight Attentional Feature Fusion: A New Baseline for Text-to-Video Retrieval](https://mlsanity.com/paper/2112.01832/) (2021)
- [Multi-Modal Multi-Instance Learning for Retinal Disease Recognition](https://mlsanity.com/paper/2109.12307/) (2021)
- [What Matters for Ad-hoc Video Search? A Large-scale Evaluation on TRECVID](https://mlsanity.com/paper/2109.01774/) (2021)

## Built on by (most cited first, 16 of 16)

- [Text Is MASS: Modeling as Stochastic Embedding for Text-Video Retrieval](https://mlsanity.com/paper/2403.17998/) (2024)
- [Know "No" Better: A Data-Driven Approach for Enhancing Negation Awareness in CLIP](https://mlsanity.com/paper/2501.10913/) (2025)
- [iSEARLE: Improving Textual Inversion for Zero-Shot Composed Image Retrieval](https://mlsanity.com/paper/2405.02951/) (2024)
- [Dual-Modal Attention-Enhanced Text-Video Retrieval with Triplet Partial Margin Contrastive Learning](https://mlsanity.com/paper/2309.11082/) (2023)
- [NeIn: Telling What You Don't Want](https://mlsanity.com/paper/2409.06481/) (2024)
- [Renmin University of China at TRECVID 2022: Improving Video Search by Feature Fusion and Negation Understanding](https://mlsanity.com/paper/2211.15039/) (2022)
- [Multimodal LLM-based Query Paraphrasing for Video Search](https://mlsanity.com/paper/2407.12341/) (2024)
- [Beyond Coarse-Grained Matching in Video-Text Retrieval](https://mlsanity.com/paper/2410.12407/) (2024)
- [From No to Know: Taxonomy, Challenges, and Opportunities for Negation Understanding in Multimodal Foundation Models](https://mlsanity.com/paper/2502.09645/) (2025)
- [Leveraging Auxiliary Information in Text-to-Video Retrieval: A Review](https://mlsanity.com/paper/2505.23952/) (2025)
- [Towards Effective Negation Modeling in Joint Audio-Text Models for Music](https://mlsanity.com/paper/2601.13931/) (2026)
- [Omni-NegCLIP: Enhancing CLIP with Front-Layer Contrastive Fine-Tuning for Comprehensive Negation Understanding](https://mlsanity.com/paper/2603.29258/) (2026)
- [Negation Beyond the Verbal Channel: Temporal Multimodal Correlates in Dialogue](https://mlsanity.com/paper/2609.16396/) (2026)
- [Adapting MLLMs for Nuanced Video Retrieval](https://mlsanity.com/paper/2512.13511/) (2025)
- [Adaptive Multi-Agent Reasoning for Text-to-Video Retrieval](https://mlsanity.com/paper/2602.19040/) (2025)
- [Understanding the Performance Plateau in Text-to-Video Retrieval: A Comprehensive Empirical and Linguistic Analysis](https://mlsanity.com/paper/2605.00826/) (2026)

## Abstract (arXiv)

Negation is a common linguistic skill that allows human to express what we do NOT want. Naturally, one might expect video retrieval to support natural-language queries with negation, e.g., finding shots of kids sitting on the floor and not playing with a dog. However, the state-of-the-art deep learning based video retrieval models lack such ability, as they are typically trained on video description datasets such as MSR-VTT and VATEX that lack negated descriptions. Their retrieved results basically ignore the negator in the sample query, incorrectly returning videos showing kids playing with dog. This paper presents the first study on learning to understand negation in video retrieval and make contributions as follows. By re-purposing two existing datasets (MSR-VTT and VATEX), we propose a new evaluation protocol for video retrieval with negation. We propose a learning based method for training a negation-aware video retrieval model. The key idea is to first construct a soft negative caption for a specific training video by partially negating its original caption, and then compute a bidirectionally constrained loss on the triplet. This auxiliary loss is weightedly added to a standard retrieval loss. Experiments on the re-purposed benchmarks show that re-training the CLIP (Contrastive Language-Image Pre-Training) model by the proposed method clearly improves its ability to handle queries with negation. In addition, the model performance on the original benchmarks is also improved.