# RECLIP: Resource-efficient CLIP by Training with Small Images

arXiv: https://arxiv.org/abs/2304.06028 · HTML: https://mlsanity.com/paper/2304.06028/

## Key facts

- Cited by 16 papers in the map (data through 14 Oct 2026).
- Builds on 48 of its 60 references that are in the map.
- First posted on arXiv 12 Apr 2023.

## Builds on (most cited first, 30 of 48)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Representation Learning with Contrastive Predictive Coding](https://mlsanity.com/paper/1807.03748/) (2018)
- [Momentum Contrast for Unsupervised Visual Representation Learning](https://mlsanity.com/paper/1911.05722/) (2019)
- [Masked Autoencoders Are Scalable Vision Learners](https://mlsanity.com/paper/2111.06377/) (2021)
- [Zero-Shot Text-to-Image Generation](https://mlsanity.com/paper/2102.12092/) (2021)
- [Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision](https://mlsanity.com/paper/2102.05918/) (2021)
- [Sigmoid Loss for Language Image Pre-Training](https://mlsanity.com/paper/2303.15343/) (2023)
- [BEiT: BERT Pre-Training of Image Transformers](https://mlsanity.com/paper/2106.08254/) (2021)
- [Learning to Prompt for Vision-Language Models](https://mlsanity.com/paper/2109.01134/) (2021)
- [The Many Faces of Robustness: A Critical Analysis of Out-of-Distribution Generalization](https://mlsanity.com/paper/2006.16241/) (2020)
- [LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairs](https://mlsanity.com/paper/2111.02114/) (2021)
- [Do ImageNet Classifiers Generalize to ImageNet?](https://mlsanity.com/paper/1902.10811/) (2019)
- [Microsoft COCO Captions: Data Collection and Evaluation Server](https://mlsanity.com/paper/1504.00325/) (2015)
- [MLP-Mixer: An all-MLP Architecture for Vision](https://mlsanity.com/paper/2105.01601/) (2021)
- [Natural Adversarial Examples](https://mlsanity.com/paper/1907.07174/) (2019)
- [Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Models](https://mlsanity.com/paper/1505.04870/) (2015)
- [CoCa: Contrastive Captioners are Image-Text Foundation Models](https://mlsanity.com/paper/2205.01917/) (2022)
- [Swin Transformer V2: Scaling Up Capacity and Resolution](https://mlsanity.com/paper/2111.09883/) (2021)
- [LVIS: A Dataset for Large Vocabulary Instance Segmentation](https://mlsanity.com/paper/1908.03195/) (2019)
- [A Closer Look at Spatiotemporal Convolutions for Action Recognition](https://mlsanity.com/paper/1711.11248/) (2017)
- [Learning Robust Global Representations by Penalizing Local Predictive Power](https://mlsanity.com/paper/1905.13549/) (2019)
- [Revisiting Unreasonable Effectiveness of Data in Deep Learning Era](https://mlsanity.com/paper/1707.02968/) (2017)
- [Open-vocabulary Object Detection via Vision and Language Knowledge Distillation](https://mlsanity.com/paper/2104.13921/) (2021)
- [PaLI: A Jointly-Scaled Multilingual Language-Image Model](https://mlsanity.com/paper/2209.06794/) (2022)

## Built on by (most cited first, 16 of 16)

- [An Inverse Scaling Law for CLIP Training](https://mlsanity.com/paper/2305.07017/) (2023)
- [ViTamin: Designing Scalable Vision Models in the Vision-Language Era](https://mlsanity.com/paper/2404.02132/) (2024)
- [Contrastive Feature Masking Open-Vocabulary Vision Transformer](https://mlsanity.com/paper/2309.00775/) (2023)
- [X-Transfer Attacks: Towards Super Transferable Adversarial Attacks on CLIP](https://mlsanity.com/paper/2505.05528/) (2025)
- [Less is More: Removing Text-regions Improves CLIP Training Efficiency and Robustness](https://mlsanity.com/paper/2305.05095/) (2023)
- [OpenVision: A Fully-Open, Cost-Effective Family of Advanced Vision Encoders for Multimodal Learning](https://mlsanity.com/paper/2505.04601/) (2025)
- [M2-Encoder: Advancing Bilingual Image-Text Understanding by Large-scale Efficient Pretraining](https://mlsanity.com/paper/2401.15896/) (2024)
- [LR0.FM: Low-Res Benchmark and Improving Robustness for Zero-Shot Classification in Foundation Models](https://mlsanity.com/paper/2502.03950/) (2025)
- [FastDINOv2: Frequency Based Curriculum Learning Improves Robustness and Training Speed](https://mlsanity.com/paper/2507.03779/) (2025)
- [From Coarse to Fine: Efficient Training for Audio Spectrogram Transformers](https://mlsanity.com/paper/2401.08415/) (2024)
- [Frequency Is What You Need: Considering Word Frequency When Text Masking Benefits Vision-Language Model Pre-training](https://mlsanity.com/paper/2412.16148/) (2024)
- [Progressive Growing of Patch Size: Curriculum Learning for Accelerated and Improved Medical Image Segmentation](https://mlsanity.com/paper/2510.23241/) (2025)
- [CanvOI, an Oncology Intelligence Foundation Model: Scaling FLOPS Differently](https://mlsanity.com/paper/2409.02885/) (2024)
- [Enhancing Vision-Language Model Pre-training with Image-text Pair Pruning Based on Word Frequency](https://mlsanity.com/paper/2410.10879/) (2024)
- [Using Knowledge Graphs to harvest datasets for efficient CLIP model training](https://mlsanity.com/paper/2505.02746/) (2025)
- [Dynamic Cluster Data Sampling for Efficient and Long-Tail-Aware Vision-Language Pre-training](https://mlsanity.com/paper/2604.27932/) (2026)

## Abstract (arXiv)

We present RECLIP (Resource-efficient CLIP), a simple method that minimizes computational resource footprint for CLIP (Contrastive Language Image Pretraining). Inspired by the notion of coarse-to-fine in computer vision, we leverage small images to learn from large-scale language supervision efficiently, and finetune the model with high-resolution data in the end. Since the complexity of the vision transformer heavily depends on input image size, our approach significantly reduces the training resource requirements both in theory and in practice. Using the same batch size and training epoch, RECLIP achieves highly competitive zero-shot classification and image-text retrieval accuracy with 6 to 8x less computational resources and 7 to 9x fewer FLOPs than the baseline. Compared to the state-of-the-art contrastive learning methods, RECLIP demonstrates 5 to 59x training resource savings while maintaining highly competitive zero-shot classification and retrieval performance. Finally, RECLIP matches the state of the art in transfer learning to open-vocabulary detection tasks, achieving 32 APr on LVIS. We hope this work will pave the path for the broader research community to explore language supervised pretraining in resource-friendly settings.