# Less is More: Removing Text-regions Improves CLIP Training Efficiency and Robustness

arXiv: https://arxiv.org/abs/2305.05095 · HTML: https://mlsanity.com/paper/2305.05095/

## Key facts

- Cited by 19 papers in the map (data through 14 Oct 2026).
- Builds on 31 of its 37 references that are in the map.
- First posted on arXiv 8 May 2023.

## Builds on (most cited first, 30 of 31)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision](https://mlsanity.com/paper/2102.05918/) (2021)
- [Sigmoid Loss for Language Image Pre-Training](https://mlsanity.com/paper/2303.15343/) (2023)
- [Training Compute-Optimal Large Language Models](https://mlsanity.com/paper/2203.15556/) (2022)
- [CoCa: Contrastive Captioners are Image-Text Foundation Models](https://mlsanity.com/paper/2205.01917/) (2022)
- [Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Concepts](https://mlsanity.com/paper/2102.08981/) (2021)
- [Scaling Language Models: Methods, Analysis & Insights from Training Gopher](https://mlsanity.com/paper/2112.11446/) (2021)
- [Scaling Vision Transformers](https://mlsanity.com/paper/2106.04560/) (2021)
- [FILIP: Fine-grained Interactive Language-Image Pre-Training](https://mlsanity.com/paper/2111.07783/) (2021)
- [Beyond neural scaling laws: beating power law scaling via data pruning](https://mlsanity.com/paper/2206.14486/) (2022)
- [Symbolic Discovery of Optimization Algorithms](https://mlsanity.com/paper/2302.06675/) (2023)
- [Supervision Exists Everywhere: A Data Efficient Contrastive Language-Image Pre-training Paradigm](https://mlsanity.com/paper/2110.05208/) (2021)
- [Scaling Language-Image Pre-training via Masking](https://mlsanity.com/paper/2212.00794/) (2022)
- [TrOCR: Transformer-based Optical Character Recognition with Pre-trained Models](https://mlsanity.com/paper/2109.10282/) (2021)
- [Character Region Awareness for Text Detection](https://mlsanity.com/paper/1904.01941/) (2019)
- [Combined Scaling for Zero-shot Transfer Learning](https://mlsanity.com/paper/2111.10050/) (2021)
- [Towards End-to-End Unified Scene Text Detection and Layout Analysis](https://mlsanity.com/paper/2203.15143/) (2022)
- [TAP: Text-Aware Pre-training for Text-VQA and Text-Caption](https://mlsanity.com/paper/2012.04638/) (2020)
- [Real-Time Scene Text Detection with Differentiable Binarization and Adaptive Scale Fusion](https://mlsanity.com/paper/2202.10304/) (2022)
- [Filtering, Distillation, and Hard Negatives for Vision-Language Pre-Training](https://mlsanity.com/paper/2301.02280/) (2023)
- [Language Matters: A Weakly Supervised Vision-Language Pre-training Approach for Scene Text Detection and Spotting](https://mlsanity.com/paper/2203.03911/) (2022)
- [PreSTU: Pre-Training for Scene-Text Understanding](https://mlsanity.com/paper/2209.05534/) (2022)
- [Non-Contrastive Learning Meets Language-Image Pre-Training](https://mlsanity.com/paper/2210.09304/) (2022)
- [STAIR: Learning Sparse Text and Image Representation in Grounded Tokens](https://mlsanity.com/paper/2301.13081/) (2023)
- [Vision-Language Pre-Training for Boosting Scene Text Detectors](https://mlsanity.com/paper/2204.13867/) (2022)
- [RECLIP: Resource-efficient CLIP by Training with Small Images](https://mlsanity.com/paper/2304.06028/) (2023)
- [Unsupervised Data Selection via Discrete Speech Representation for ASR](https://mlsanity.com/paper/2204.01981/) (2022)

## Built on by (most cited first, 19 of 19)

- [Multimodal Foundation Models: From Specialists to General-Purpose Assistants](https://mlsanity.com/paper/2309.10020/) (2023)
- [Improving Multimodal Datasets with Image Captioning](https://mlsanity.com/paper/2307.10350/) (2023)
- [VeCLIP: Improving CLIP Training via Visual-enriched Captions](https://mlsanity.com/paper/2310.07699/) (2023)
- [Data curation via joint example selection further accelerates multimodal learning](https://mlsanity.com/paper/2406.17711/) (2024)
- [Delving into Multi-modal Multi-task Foundation Models for Road Scene Understanding: From Learning Paradigm Perspectives](https://mlsanity.com/paper/2402.02968/) (2024)
- [MLLMs-Augmented Visual-Language Representation Learning](https://mlsanity.com/paper/2311.18765/) (2023)
- [CLIP-CID: Efficient CLIP Distillation via Cluster-Instance Discrimination](https://mlsanity.com/paper/2408.09441/) (2024)
- [The Devil is in the Details: A Deep Dive into the Rabbit Hole of Data Filtering](https://mlsanity.com/paper/2309.15954/) (2023)
- [Active Data Curation Effectively Distills Large-Scale Multimodal Models](https://mlsanity.com/paper/2411.18674/) (2024)
- [Scaling Pre-training to One Hundred Billion Data for Vision Language Models](https://mlsanity.com/paper/2502.07617/) (2025)
- [Parrot Captions Teach CLIP to Spot Text](https://mlsanity.com/paper/2312.14232/) (2023)
- [Knowledge Transfer from Vision Foundation Models for Efficient Training of Small Task-specific Models](https://mlsanity.com/paper/2311.18237/) (2023)
- [USE: Universal Segment Embeddings for Open-Vocabulary Image Segmentation](https://mlsanity.com/paper/2406.05271/) (2024)
- [ReME: A Data-Centric Framework for Training-Free Open-Vocabulary Segmentation](https://mlsanity.com/paper/2506.21233/) (2025)
- [MOFI: Learning Image Representations from Noisy Entity Annotated Images](https://mlsanity.com/paper/2306.07952/) (2023)
- [Efficient Self-Supervised Learning for Earth Observation via Dynamic Dataset Curation](https://mlsanity.com/paper/2504.06962/) (2025)
- [Focus, Distinguish, and Prompt: Unleashing CLIP for Efficient and Flexible Scene Text Retrieval](https://mlsanity.com/paper/2408.00441/) (2024)
- [DOSE: Data Selection for Multi-Modal LLMs via Off-the-Shelf Models](https://mlsanity.com/paper/2604.16979/) (2026)
- [Sparse vs Contiguous Adversarial Pixel Perturbations in Multimodal Models: An Empirical Analysis](https://mlsanity.com/paper/2407.18251/) (2024)

## Abstract (arXiv)

The CLIP (Contrastive Language-Image Pre-training) model and its variants are becoming the de facto backbone in many applications. However, training a CLIP model from hundreds of millions of image-text pairs can be prohibitively expensive. Furthermore, the conventional CLIP model doesn't differentiate between the visual semantics and meaning of text regions embedded in images. This can lead to non-robustness when the text in the embedded region doesn't match the image's visual appearance. In this paper, we discuss two effective approaches to improve the efficiency and robustness of CLIP training: (1) augmenting the training dataset while maintaining the same number of optimization steps, and (2) filtering out samples that contain text regions in the image. By doing so, we significantly improve the classification and retrieval accuracy on public benchmarks like ImageNet and CoCo. Filtering out images with text regions also protects the model from typographic attacks. To verify this, we build a new dataset named ImageNet with Adversarial Text Regions (ImageNet-Attr). Our filter-based CLIP model demonstrates a top-1 accuracy of 68.78%, outperforming previous models whose accuracy was all below 50%.