# Internet Explorer: Targeted Representation Learning on the Open Web

arXiv: https://arxiv.org/abs/2302.14051 · HTML: https://mlsanity.com/paper/2302.14051/

## Key facts

- Cited by 21 papers in the map (data through 14 Oct 2026).
- Builds on 34 of its 56 references that are in the map.
- First posted on arXiv 27 Feb 2023.

## Builds on (most cited first, 30 of 34)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [Representation Learning with Contrastive Predictive Coding](https://mlsanity.com/paper/1807.03748/) (2018)
- [Momentum Contrast for Unsupervised Visual Representation Learning](https://mlsanity.com/paper/1911.05722/) (2019)
- [Masked Autoencoders Are Scalable Vision Learners](https://mlsanity.com/paper/2111.06377/) (2021)
- [Emerging Properties in Self-Supervised Vision Transformers](https://mlsanity.com/paper/2104.14294/) (2021)
- [Bootstrap your own latent: A new approach to self-supervised Learning](https://mlsanity.com/paper/2006.07733/) (2020)
- [LAION-5B: An open large-scale dataset for training next generation image-text models](https://mlsanity.com/paper/2210.08402/) (2022)
- [FaceNet: A Unified Embedding for Face Recognition and Clustering](https://mlsanity.com/paper/1503.03832/) (2015)
- [Billion-scale similarity search with GPUs](https://mlsanity.com/paper/1702.08734/) (2017)
- [BEiT: BERT Pre-Training of Image Transformers](https://mlsanity.com/paper/2106.08254/) (2021)
- [Barlow Twins: Self-Supervised Learning via Redundancy Reduction](https://mlsanity.com/paper/2103.03230/) (2021)
- [Understanding Black-box Predictions via Influence Functions](https://mlsanity.com/paper/1703.04730/) (2017)
- [An Empirical Study of Training Self-Supervised Vision Transformers](https://mlsanity.com/paper/2104.02057/) (2021)
- [LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairs](https://mlsanity.com/paper/2111.02114/) (2021)
- [Contrastive Multiview Coding](https://mlsanity.com/paper/1906.05849/) (2019)
- [VICReg: Variance-Invariance-Covariance Regularization for Self-Supervised Learning](https://mlsanity.com/paper/2105.04906/) (2021)
- [Contrastive Learning with Hard Negative Samples](https://mlsanity.com/paper/2010.04592/) (2020)
- [Exploring the Limits of Weakly Supervised Pretraining](https://mlsanity.com/paper/1805.00932/) (2018)
- [Large Batch Training of Convolutional Networks](https://mlsanity.com/paper/1708.03888/) (2017)
- [Deep Learning on a Data Diet: Finding Important Examples Early in Training](https://mlsanity.com/paper/2107.07075/) (2021)
- [What Neural Networks Memorize and Why: Discovering the Long Tail via Influence Estimation](https://mlsanity.com/paper/2008.03703/) (2020)
- [Deep Metric Learning via Lifted Structured Feature Embedding](https://mlsanity.com/paper/1511.06452/) (2015)
- [Functional Map of the World](https://mlsanity.com/paper/1711.07846/) (2017)
- [Sampling Matters in Deep Embedding Learning](https://mlsanity.com/paper/1706.07567/) (2017)
- [Datamodels: Predicting Predictions from Training Data](https://mlsanity.com/paper/2202.00622/) (2022)
- [Deep Metric Learning with Hierarchical Triplet Loss](https://mlsanity.com/paper/1810.06951/) (2018)
- [Smart Mining for Deep Metric Learning](https://mlsanity.com/paper/1704.01285/) (2017)

## Built on by (most cited first, 21 of 21)

- [Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMs](https://mlsanity.com/paper/2406.16860/) (2024)
- [SynthCLIP: Are We Ready for a Fully Synthetic CLIP Training?](https://mlsanity.com/paper/2402.01832/) (2024)
- [The Neglected Tails in Vision-Language Models](https://mlsanity.com/paper/2401.12425/) (2024)
- [Large Language Models as Tax Attorneys: A Case Study in Legal Capabilities Emergence](https://mlsanity.com/paper/2306.07075/) (2023)
- [MMEvol: Empowering Multimodal Large Language Models with Evol-Instruct](https://mlsanity.com/paper/2409.05840/) (2024)
- [V-IRL: Grounding Virtual Intelligence in Real Life](https://mlsanity.com/paper/2402.03310/) (2024)
- [Interactive Natural Language Processing](https://mlsanity.com/paper/2305.13246/) (2023)
- [Knowledge Transfer from Vision Foundation Models for Efficient Training of Small Task-specific Models](https://mlsanity.com/paper/2311.18237/) (2023)
- [Few-Shot Recognition via Stage-Wise Retrieval-Augmented Finetuning](https://mlsanity.com/paper/2406.11148/) (2024)
- [Image retrieval outperforms diffusion models on data augmentation](https://mlsanity.com/paper/2304.10253/) (2023)
- [Deceptive Alignment Monitoring](https://mlsanity.com/paper/2307.10569/) (2023)
- [Learning from Neighbors: Category Extrapolation for Long-Tail Learning](https://mlsanity.com/paper/2410.15980/) (2024)
- [On Pretraining Data Diversity for Self-Supervised Learning](https://mlsanity.com/paper/2403.13808/) (2024)
- [GRID: A Platform for General Robot Intelligence Development](https://mlsanity.com/paper/2310.00887/) (2023)
- [From Categories to Classifiers: Name-Only Continual Learning by Exploring the Web](https://mlsanity.com/paper/2311.11293/) (2023)
- [Active Learning via Vision-Language Model Adaptation with Open Data](https://mlsanity.com/paper/2506.01724/) (2025)
- [Enabling Validation for Robust Few-Shot Recognition](https://mlsanity.com/paper/2506.04713/) (2025)
- [Precision at Scale: Domain-Specific Datasets On-Demand](https://mlsanity.com/paper/2407.03463/) (2024)
- [MYCROFT: Towards Effective and Efficient External Data Augmentation](https://mlsanity.com/paper/2410.08432/) (2024)
- [Zero-shot Retrieval: Augmenting Pre-trained Models with Search Engines](https://mlsanity.com/paper/2311.17949/) (2023)
- [On the Importance of Pretraining Data Alignment for Atomic Property Prediction](https://mlsanity.com/paper/2502.11085/) (2025)

## Abstract (arXiv)

Modern vision models typically rely on fine-tuning general-purpose models pre-trained on large, static datasets. These general-purpose models only capture the knowledge within their pre-training datasets, which are tiny, out-of-date snapshots of the Internet -- where billions of images are uploaded each day. We suggest an alternate approach: rather than hoping our static datasets transfer to our desired tasks after large-scale pre-training, we propose dynamically utilizing the Internet to quickly train a small-scale model that does extremely well on the task at hand. Our approach, called Internet Explorer, explores the web in a self-supervised manner to progressively find relevant examples that improve performance on a desired target dataset. It cycles between searching for images on the Internet with text queries, self-supervised training on downloaded images, determining which images were useful, and prioritizing what to search for next. We evaluate Internet Explorer across several datasets and show that it outperforms or matches CLIP oracle performance by using just a single GPU desktop to actively query the Internet for 30--40 hours. Results, visualizations, and videos at https://internet-explorer-ssl.github.io/