# Lightweight Attentional Feature Fusion: A New Baseline for Text-to-Video Retrieval

arXiv: https://arxiv.org/abs/2112.01832 · HTML: https://mlsanity.com/paper/2112.01832/

## Key facts

- Cited by 23 papers in the map (data through 14 Oct 2026).
- Builds on 33 of its 68 references that are in the map.
- First posted on arXiv 3 Dec 2021.

## Builds on (most cited first, 30 of 33)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [CBAM: Convolutional Block Attention Module](https://mlsanity.com/paper/1807.06521/) (2018)
- [Is Space-Time Attention All You Need for Video Understanding?](https://mlsanity.com/paper/2102.05095/) (2021)
- [Frozen in Time: A Joint Video and Image Encoder for End-to-End Retrieval](https://mlsanity.com/paper/2104.00650/) (2021)
- [Attention-based Deep Multiple Instance Learning](https://mlsanity.com/paper/1802.04712/) (2018)
- [Multimodal Machine Learning: A Survey and Taxonomy](https://mlsanity.com/paper/1705.09406/) (2017)
- [Language Modeling with Gated Convolutional Networks](https://mlsanity.com/paper/1612.08083/) (2016)
- [CLIP4Clip: An Empirical Study of CLIP for End to End Video Clip Retrieval](https://mlsanity.com/paper/2104.08860/) (2021)
- [X3D: Expanding Architectures for Efficient Video Recognition](https://mlsanity.com/paper/2004.04730/) (2020)
- [Exploring the Limits of Weakly Supervised Pretraining](https://mlsanity.com/paper/1805.00932/) (2018)
- [VATEX: A Large-Scale, High-Quality Multilingual Dataset for Video-and-Language Research](https://mlsanity.com/paper/1904.03493/) (2019)
- [Multi-modal Transformer for Video Retrieval](https://mlsanity.com/paper/2007.10639/) (2020)
- [CLIP2Video: Mastering Video-Text Retrieval via Image CLIP](https://mlsanity.com/paper/2106.11097/) (2021)
- [Use What You Have: Video Retrieval Using Representations From Collaborative Experts](https://mlsanity.com/paper/1907.13487/) (2019)
- [A Joint Sequence Fusion Model for Video Question Answering and Retrieval](https://mlsanity.com/paper/1808.02559/) (2018)
- [Fine-grained Video-Text Retrieval with Hierarchical Graph Reasoning](https://mlsanity.com/paper/2003.00392/) (2020)
- [Support-set bottlenecks for video-text representation learning](https://mlsanity.com/paper/2010.02824/) (2020)
- [Learning Visual Features from Large Weakly Supervised Data](https://mlsanity.com/paper/1511.02251/) (2015)
- [Large-scale weakly-supervised pre-training for video action recognition](https://mlsanity.com/paper/1905.00561/) (2019)
- [TGIF: A New Dataset and Benchmark on Animated GIF Description](https://mlsanity.com/paper/1604.02748/) (2016)
- [Learning a Text-Video Embedding from Incomplete and Heterogeneous Data](https://mlsanity.com/paper/1804.02516/) (2018)
- [Fine-Grained Action Retrieval Through Multiple Parts-of-Speech Embeddings](https://mlsanity.com/paper/1908.03477/) (2019)
- [Polysemous Visual-Semantic Embedding for Cross-Modal Retrieval](https://mlsanity.com/paper/1906.04402/) (2019)
- [TEACHTEXT: CrossModal Generalized Distillation for Text-Video Retrieval](https://mlsanity.com/paper/2104.08271/) (2021)
- [Noise Estimation Using Density Estimation for Self-Supervised Multimodal Learning](https://mlsanity.com/paper/2003.03186/) (2020)
- [A Straightforward Framework For Video Retrieval Using CLIP](https://mlsanity.com/paper/2102.12443/) (2021)
- [Predicting Visual Features from Text for Image and Video Caption Retrieval](https://mlsanity.com/paper/1709.01362/) (2017)

## Built on by (most cited first, 23 of 23)

- [Koala: Key frame-conditioned long video-LLM](https://mlsanity.com/paper/2404.04346/) (2024)
- [A Review of Deep Learning for Video Captioning](https://mlsanity.com/paper/2304.11431/) (2023)
- [Audio-Enhanced Text-to-Video Retrieval using Text-Conditioned Feature Alignment](https://mlsanity.com/paper/2307.12964/) (2023)
- [Learn to Understand Negation in Video Retrieval](https://mlsanity.com/paper/2205.00132/) (2022)
- [Towards Efficient and Effective Text-to-Video Retrieval with Coarse-to-Fine Visual Representation Learning](https://mlsanity.com/paper/2401.00701/) (2024)
- [CL2CM: Improving Cross-Lingual Cross-Modal Retrieval via Cross-Lingual Knowledge Transfer](https://mlsanity.com/paper/2312.08984/) (2023)
- [PRVR: Partially Relevant Video Retrieval](https://mlsanity.com/paper/2208.12510/) (2022)
- [Vision-Language Models Learn Super Images for Efficient Partially Relevant Video Retrieval](https://mlsanity.com/paper/2312.00414/) (2023)
- [Hybrid-Tower: Fine-grained Pseudo-query Interaction and Generation for Text-to-Video Retrieval](https://mlsanity.com/paper/2509.04773/) (2025)
- [(Un)likelihood Training for Interpretable Embedding](https://mlsanity.com/paper/2207.00282/) (2022)
- [Improving Interpretable Embeddings for Ad-hoc Video Search with Generative Captions and Multi-word Concept Bank](https://mlsanity.com/paper/2404.06173/) (2024)
- [Dual Learning with Dynamic Knowledge Distillation and Soft Alignment for Partially Relevant Video Retrieval](https://mlsanity.com/paper/2510.12283/) (2025)
- [Renmin University of China at TRECVID 2022: Improving Video Search by Feature Fusion and Negation Understanding](https://mlsanity.com/paper/2211.15039/) (2022)
- [TeachCLIP: Multi-Grained Teaching for Efficient Text-to-Video Retrieval](https://mlsanity.com/paper/2308.01217/) (2023)
- [Video Editing for Video Retrieval](https://mlsanity.com/paper/2402.02335/) (2024)
- [Multimodal LLM-based Query Paraphrasing for Video Search](https://mlsanity.com/paper/2407.12341/) (2024)
- [Beyond Coarse-Grained Matching in Video-Text Retrieval](https://mlsanity.com/paper/2410.12407/) (2024)
- [Leveraging Auxiliary Information in Text-to-Video Retrieval: A Review](https://mlsanity.com/paper/2505.23952/) (2025)
- [SpatialMem: Metric-Aligned Long-Horizon Video Memory for Language Grounding and QA](https://mlsanity.com/paper/2601.14895/) (2026)
- [Adaptive Multi-Agent Reasoning for Text-to-Video Retrieval](https://mlsanity.com/paper/2602.19040/) (2025)
- [SAVE: Speech-Aware Video Representation Learning for Video-Text Retrieval](https://mlsanity.com/paper/2603.08224/) (2026)
- [Fusion Learning from Dynamic Functional Connectivity: Combining the Amplitude and Phase of fMRI Signals to Identify Brain Disorders](https://mlsanity.com/paper/2603.24603/) (2026)
- [EagleNet: Energy-Aware Fine-Grained Relationship Learning Network for Text-Video Retrieval](https://mlsanity.com/paper/2603.25267/) (2026)

## Abstract (arXiv)

In this paper we revisit feature fusion, an old-fashioned topic, in the new context of text-to-video retrieval. Different from previous research that considers feature fusion only at one end, let it be video or text, we aim for feature fusion for both ends within a unified framework. We hypothesize that optimizing the convex combination of the features is preferred to modeling their correlations by computationally heavy multi-head self attention. We propose Lightweight Attentional Feature Fusion (LAFF). LAFF performs feature fusion at both early and late stages and at both video and text ends, making it a powerful method for exploiting diverse (off-the-shelf) features. The interpretability of LAFF can be used for feature selection. Extensive experiments on five public benchmark sets (MSR-VTT, MSVD, TGIF, VATEX and TRECVID AVS 2016-2020) justify LAFF as a new baseline for text-to-video retrieval.