# From Categories to Classifiers: Name-Only Continual Learning by Exploring the Web

arXiv: https://arxiv.org/abs/2311.11293 · HTML: https://mlsanity.com/paper/2311.11293/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 50 of its 75 references that are in the map.
- First posted on arXiv 19 Nov 2023.

## Builds on (most cited first, 30 of 50)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [DINOv2: Learning Robust Visual Features without Supervision](https://mlsanity.com/paper/2304.07193/) (2023)
- [Hierarchical Text-Conditional Image Generation with CLIP Latents](https://mlsanity.com/paper/2204.06125/) (2022)
- [LAION-5B: An open large-scale dataset for training next generation image-text models](https://mlsanity.com/paper/2210.08402/) (2022)
- [Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision](https://mlsanity.com/paper/2102.05918/) (2021)
- [iCaRL: Incremental Classifier and Representation Learning](https://mlsanity.com/paper/1611.07725/) (2016)
- [Fine-Grained Visual Classification of Aircraft](https://mlsanity.com/paper/1306.5151/) (2013)
- [An Empirical Study of Training Self-Supervised Vision Transformers](https://mlsanity.com/paper/2104.02057/) (2021)
- [Reproducible scaling laws for contrastive language-image learning](https://mlsanity.com/paper/2212.07143/) (2022)
- [Deeper, Broader and Artier Domain Generalization](https://mlsanity.com/paper/1710.03077/) (2017)
- [Revisiting Unreasonable Effectiveness of Data in Deep Learning Era](https://mlsanity.com/paper/1707.02968/) (2017)
- [Efficient and robust approximate nearest neighbor search using Hierarchical Navigable Small World graphs](https://mlsanity.com/paper/1603.09320/) (2016)
- [Robust fine-tuning of zero-shot models](https://mlsanity.com/paper/2109.01903/) (2021)
- [Exploring the Limits of Weakly Supervised Pretraining](https://mlsanity.com/paper/1805.00932/) (2018)
- [Selection via Proxy: Efficient Data Selection for Deep Learning](https://mlsanity.com/paper/1906.11829/) (2019)
- [Visual Classification via Description from Large Language Models](https://mlsanity.com/paper/2210.07183/) (2022)
- [Is synthetic data from generative models ready for image recognition?](https://mlsanity.com/paper/2210.07574/) (2022)
- [Your Diffusion Model is Secretly a Zero-Shot Classifier](https://mlsanity.com/paper/2303.16203/) (2023)
- [What does a platypus look like? Generating customized prompts for zero-shot image classification](https://mlsanity.com/paper/2209.03320/) (2022)
- [WebVision Database: Visual Learning and Understanding from Web Data](https://mlsanity.com/paper/1708.02862/) (2017)
- [Large image datasets: A pyrrhic win for computer vision?](https://mlsanity.com/paper/2006.16923/) (2020)
- [Learning Visual Features from Large Weakly Supervised Data](https://mlsanity.com/paper/1511.02251/) (2015)
- [Prompt, Generate, then Cache: Cascade of Foundation Models makes Strong Few-shot Learners](https://mlsanity.com/paper/2303.02151/) (2023)
- [Mining Cross-Image Semantics for Weakly Supervised Semantic Segmentation](https://mlsanity.com/paper/2007.01947/) (2020)
- [Large-scale weakly-supervised pre-training for video action recognition](https://mlsanity.com/paper/1905.00561/) (2019)
- [SuS-X: Training-Free Name-Only Transfer of Vision-Language Models](https://mlsanity.com/paper/2211.16198/) (2022)
- [Revisiting Weakly Supervised Pre-Training of Visual Perception Models](https://mlsanity.com/paper/2201.08371/) (2022)
- [CALIP: Zero-Shot Enhancement of CLIP with Parameter-free Attention](https://mlsanity.com/paper/2209.14169/) (2022)
- [Waffling around for Performance: Visual Classification with Random Words and Broad Concepts](https://mlsanity.com/paper/2306.07282/) (2023)

## Built on by (most cited first, 5 of 5)

- [No "Zero-Shot" Without Exponential Data: Pretraining Concept Frequency Determines Multimodal Model Performance](https://mlsanity.com/paper/2404.04125/) (2024)
- [kNN-CLIP: Retrieval Enables Training-Free Segmentation on Continually Expanding Large Vocabularies](https://mlsanity.com/paper/2404.09447/) (2024)
- [Beyond Cosine Decay: On the effectiveness of Infinite Learning Rate Schedule for Continual Pre-training](https://mlsanity.com/paper/2503.02844/) (2025)
- [Premonition: Using Generative Models to Preempt Future Data Changes in Continual Learning](https://mlsanity.com/paper/2403.07356/) (2024)
- [Conformal-in-the-Loop for Learning with Imbalanced Noisy Data](https://mlsanity.com/paper/2411.02281/) (2024)

## Abstract (arXiv)

Continual Learning (CL) often relies on the availability of extensive annotated datasets, an assumption that is unrealistically time-consuming and costly in practice. We explore a novel paradigm termed name-only continual learning where time and cost constraints prohibit manual annotation. In this scenario, learners adapt to new category shifts using only category names without the luxury of annotated training data. Our proposed solution leverages the expansive and ever-evolving internet to query and download uncurated webly-supervised data for image classification. We investigate the reliability of our web data and find them comparable, and in some cases superior, to manually annotated datasets. Additionally, we show that by harnessing the web, we can create support sets that surpass state-of-the-art name-only classification that create support sets using generative models or image retrieval from LAION-5B, achieving up to 25% boost in accuracy. When applied across varied continual learning contexts, our method consistently exhibits a small performance gap in comparison to models trained on manually annotated datasets. We present EvoTrends, a class-incremental dataset made from the web to capture real-world trends, created in just minutes. Overall, this paper underscores the potential of using uncurated webly-supervised data to mitigate the challenges associated with manual data labeling in continual learning.