# ALADIN: Distilling Fine-grained Alignment Scores for Efficient Image-Text Matching and Retrieval

arXiv: https://arxiv.org/abs/2207.14757 · HTML: https://mlsanity.com/paper/2207.14757/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 32 of its 43 references that are in the map.
- First posted on arXiv 29 Jul 2022.

## Builds on (most cited first, 30 of 32)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Emerging Properties in Self-Supervised Vision Transformers](https://mlsanity.com/paper/2104.14294/) (2021)
- [BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension](https://mlsanity.com/paper/1910.13461/) (2019)
- [Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision](https://mlsanity.com/paper/2102.05918/) (2021)
- [ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Tasks](https://mlsanity.com/paper/1908.02265/) (2019)
- [Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answering](https://mlsanity.com/paper/1707.07998/) (2017)
- [Self-training with Noisy Student improves ImageNet classification](https://mlsanity.com/paper/1911.04252/) (2019)
- [Oscar: Object-Semantics Aligned Pre-training for Vision-Language Tasks](https://mlsanity.com/paper/2004.06165/) (2020)
- [Deep Visual-Semantic Alignments for Generating Image Descriptions](https://mlsanity.com/paper/1412.2306/) (2014)
- [VL-BERT: Pre-training of Generic Visual-Linguistic Representations](https://mlsanity.com/paper/1908.08530/) (2019)
- [Be Your Own Teacher: Improve the Performance of Convolutional Neural Networks via Self Distillation](https://mlsanity.com/paper/1905.08094/) (2019)
- [Unified Vision-Language Pre-Training for Image Captioning and VQA](https://mlsanity.com/paper/1909.11059/) (2019)
- [Unicoder-VL: A Universal Encoder for Vision and Language by Cross-modal Pre-training](https://mlsanity.com/paper/1908.06066/) (2019)
- [Meshed-Memory Transformer for Image Captioning](https://mlsanity.com/paper/1912.08226/) (2019)
- [Visual Semantic Reasoning for Image-Text Matching](https://mlsanity.com/paper/1909.02701/) (2019)
- [Large scale distributed neural network training through online distillation](https://mlsanity.com/paper/1804.03235/) (2018)
- [From Show to Tell: A Survey on Deep Learning-based Image Captioning](https://mlsanity.com/paper/2107.06912/) (2021)
- [ImageBERT: Cross-modal Pre-training with Large-scale Weak-supervised Image-Text Data](https://mlsanity.com/paper/2001.07966/) (2020)
- [Adversarial Representation Learning for Text-to-Image Matching](https://mlsanity.com/paper/1908.10534/) (2019)
- [Fine-grained Visual Textual Alignment for Cross-Modal Retrieval using Transformer Encoders](https://mlsanity.com/paper/2008.05231/) (2020)
- [More Grounded Image Captioning by Distilling Image-Text Matching Model](https://mlsanity.com/paper/2004.00390/) (2020)
- [Learning Dual Semantic Relations with Graph Attention for Image-Text Matching](https://mlsanity.com/paper/2010.11550/) (2020)
- [Context-Aware Learning to Rank with Self-Attention](https://mlsanity.com/paper/2005.10084/) (2020)
- [Transformer Reasoning Network for Image-Text Matching and Retrieval](https://mlsanity.com/paper/2004.09144/) (2020)
- [An Alternative Cross Entropy Loss for Learning-to-Rank](https://mlsanity.com/paper/1911.09798/) (2019)
- [CaMEL: Mean Teacher Learning for Image Captioning](https://mlsanity.com/paper/2202.10492/) (2022)
- [Learning Visual Relation Priors for Image-Text Matching and Image Captioning with Neural Scene Graph Generators](https://mlsanity.com/paper/1909.09953/) (2019)
- [Weakly Supervised Relative Spatial Reasoning for Visual Question Answering](https://mlsanity.com/paper/2109.01934/) (2021)

## Built on by (most cited first, 5 of 5)

- [Is CLIP the main roadblock for fine-grained open-world perception?](https://mlsanity.com/paper/2404.03539/) (2024)
- [Text-to-Motion Retrieval: Towards Joint Understanding of Human Motion Data and Natural Language](https://mlsanity.com/paper/2305.15842/) (2023)
- [Towards Retrieval-Augmented Architectures for Image Captioning](https://mlsanity.com/paper/2405.13127/) (2024)
- [Joint-Dataset Learning and Cross-Consistent Regularization for Text-to-Motion Retrieval](https://mlsanity.com/paper/2407.02104/) (2024)
- [Maybe you are looking for CroQS: Cross-modal Query Suggestion for Text-to-Image Retrieval](https://mlsanity.com/paper/2412.13834/) (2024)

## Abstract (arXiv)

Image-text matching is gaining a leading role among tasks involving the joint understanding of vision and language. In literature, this task is often used as a pre-training objective to forge architectures able to jointly deal with images and texts. Nonetheless, it has a direct downstream application: cross-modal retrieval, which consists in finding images related to a given query text or vice-versa. Solving this task is of critical importance in cross-modal search engines. Many recent methods proposed effective solutions to the image-text matching problem, mostly using recent large vision-language (VL) Transformer networks. However, these models are often computationally expensive, especially at inference time. This prevents their adoption in large-scale cross-modal retrieval scenarios, where results should be provided to the user almost instantaneously. In this paper, we propose to fill in the gap between effectiveness and efficiency by proposing an ALign And DIstill Network (ALADIN). ALADIN first produces high-effective scores by aligning at fine-grained level images and texts. Then, it learns a shared embedding space - where an efficient kNN search can be performed - by distilling the relevance scores obtained from the fine-grained alignments. We obtained remarkable results on MS-COCO, showing that our method can compete with state-of-the-art VL Transformers while being almost 90 times faster. The code for reproducing our results is available at https://github.com/mesnico/ALADIN.