# Leaner and Faster: Two-Stage Model Compression for Lightweight Text-Image Retrieval

arXiv: https://arxiv.org/abs/2204.13913 · HTML: https://mlsanity.com/paper/2204.13913/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 19 of its 20 references that are in the map.
- First posted on arXiv 29 Apr 2022.

## Builds on (most cited first, 19 of 19)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks](https://mlsanity.com/paper/1506.01497/) (2015)
- [Representation Learning with Contrastive Predictive Coding](https://mlsanity.com/paper/1807.03748/) (2018)
- [Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision](https://mlsanity.com/paper/2102.05918/) (2021)
- [VQA: Visual Question Answering](https://mlsanity.com/paper/1505.00468/) (2015)
- [Billion-scale similarity search with GPUs](https://mlsanity.com/paper/1702.08734/) (2017)
- [SimCSE: Simple Contrastive Learning of Sentence Embeddings](https://mlsanity.com/paper/2104.08821/) (2021)
- [Decoupled Weight Decay Regularization](https://mlsanity.com/paper/1711.05101/) (2017)
- [LXMERT: Learning Cross-Modality Encoder Representations from Transformers](https://mlsanity.com/paper/1908.07490/) (2019)
- [Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Models](https://mlsanity.com/paper/1505.04870/) (2015)
- [Oscar: Object-Semantics Aligned Pre-training for Vision-Language Tasks](https://mlsanity.com/paper/2004.06165/) (2020)
- [TinyBERT: Distilling BERT for Natural Language Understanding](https://mlsanity.com/paper/1909.10351/) (2019)
- [ImageNet-21K Pretraining for the Masses](https://mlsanity.com/paper/2104.10972/) (2021)
- [A Corpus for Reasoning About Natural Language Grounded in Photographs](https://mlsanity.com/paper/1811.00491/) (2018)
- [UNITER: UNiversal Image-TExt Representation Learning](https://mlsanity.com/paper/1909.11740/) (2019)

## Built on by (most cited first, 4 of 4)

- [ConaCLIP: Exploring Distillation of Fully-Connected Knowledge Interaction Graph for Lightweight Text-Image Retrieval](https://mlsanity.com/paper/2305.17652/) (2023)
- [MOSAIC: Multi-Object Segmented Arbitrary Stylization Using CLIP](https://mlsanity.com/paper/2309.13716/) (2023)
- [The Use of Multimodal Large Language Models to Detect Objects from Thermal Images: Transportation Applications](https://mlsanity.com/paper/2406.13898/) (2024)
- [MCAD: Multi-teacher Cross-modal Alignment Distillation for efficient image-text retrieval](https://mlsanity.com/paper/2310.19654/) (2023)

## Abstract (arXiv)

Current text-image approaches (e.g., CLIP) typically adopt dual-encoder architecture using pre-trained vision-language representation. However, these models still pose non-trivial memory requirements and substantial incremental indexing time, which makes them less practical on mobile devices. In this paper, we present an effective two-stage framework to compress large pre-trained dual-encoder for lightweight text-image retrieval. The resulting model is smaller (39% of the original), faster (1.6x/2.9x for processing image/text respectively), yet performs on par with or better than the original full model on Flickr30K and MSCOCO benchmarks. We also open-source an accompanying realistic mobile image search application.