# Concept Generalization in Visual Representation Learning

arXiv: https://arxiv.org/abs/2012.05649 · HTML: https://mlsanity.com/paper/2012.05649/

## Key facts

- Cited by 35 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 10 Dec 2020.

## Built on by (most cited first, 30 of 35)

- [Emerging Properties in Self-Supervised Vision Transformers](https://mlsanity.com/paper/2104.14294/) (2021)
- [Open-Set Recognition: a Good Closed-Set Classifier is All You Need?](https://mlsanity.com/paper/2110.06207/) (2021)
- [Neural Feature Fusion Fields: 3D Distillation of Self-Supervised 2D Image Representations](https://mlsanity.com/paper/2209.03494/) (2022)
- [Fake it till you make it: Learning transferable representations from synthetic ImageNet clones](https://mlsanity.com/paper/2212.08420/) (2022)
- [Revealing the Dark Secrets of Masked Image Modeling](https://mlsanity.com/paper/2205.13543/) (2022)
- [When Does Contrastive Visual Representation Learning Work?](https://mlsanity.com/paper/2105.05837/) (2021)
- [Vision Models Are More Robust And Fair When Pretrained On Uncurated Images Without Supervision](https://mlsanity.com/paper/2202.08360/) (2022)
- [PASS: An ImageNet replacement for self-supervised pretraining without humans](https://mlsanity.com/paper/2109.13228/) (2021)
- [ProposalCLIP: Unsupervised Open-Category Object Proposal Generation via Exploiting CLIP Cues](https://mlsanity.com/paper/2201.06696/) (2022)
- [Benchmarking Omni-Vision Representation through the Lens of Visual Realms](https://mlsanity.com/paper/2207.07106/) (2022)
- [HumanBench: Towards General Human-centric Perception with Projector Assisted Pretraining](https://mlsanity.com/paper/2303.05675/) (2023)
- [Contrasting Contrastive Self-Supervised Representation Learning Pipelines](https://mlsanity.com/paper/2103.14005/) (2021)
- [Revisiting the Transferability of Supervised Pretraining: an MLP Perspective](https://mlsanity.com/paper/2112.00496/) (2021)
- [Dataset Inference for Self-Supervised Models](https://mlsanity.com/paper/2209.09024/) (2022)
- [African or European Swallow? Benchmarking Large Vision-Language Models for Fine-Grained Object Classification](https://mlsanity.com/paper/2406.14496/) (2024)
- [UNIC: Universal Classification Models via Multi-teacher Distillation](https://mlsanity.com/paper/2408.05088/) (2024)
- [How Severe is Benchmark-Sensitivity in Video Self-Supervised Learning?](https://mlsanity.com/paper/2203.14221/) (2022)
- [No Reason for No Supervision: Improved Generalization in Supervised Models](https://mlsanity.com/paper/2206.15369/) (2022)
- [Discriminability-Transferability Trade-Off: An Information-Theoretic Perspective](https://mlsanity.com/paper/2203.03871/) (2022)
- [Continual Learning with Evolving Class Ontologies](https://mlsanity.com/paper/2210.04993/) (2022)
- [On Label Granularity and Object Localization](https://mlsanity.com/paper/2207.10225/) (2022)
- [Domain Invariant Masked Autoencoders for Self-supervised Learning from Multi-domains](https://mlsanity.com/paper/2205.04771/) (2022)
- [Measuring the Interpretability of Unsupervised Representations via Quantized Reverse Probing](https://mlsanity.com/paper/2209.03268/) (2022)
- [Granularity-aware Adaptation for Image Retrieval over Multiple Tasks](https://mlsanity.com/paper/2210.02254/) (2022)
- [DPStyler: Dynamic PromptStyler for Source-Free Domain Generalization](https://mlsanity.com/paper/2403.16697/) (2024)
- [Empirical Evaluation and Theoretical Analysis for Representation Learning: A Survey](https://mlsanity.com/paper/2204.08226/) (2022)
- [Revisiting Supervision for Continual Representation Learning](https://mlsanity.com/paper/2311.13321/) (2023)
- [Efficient unsupervised domain adaptation via self-supervised vision transformer and synergistic cross-domain alignment](https://mlsanity.com/paper/2407.21311/) (2024)
- [Towards Realistic Zero-Shot Classification via Self Structural Semantic Alignment](https://mlsanity.com/paper/2308.12960/) (2023)
- [SEVERE++: Evaluating Benchmark Sensitivity in Generalization of Video Representation Learning](https://mlsanity.com/paper/2504.05706/) (2025)

## Abstract (arXiv)

Measuring concept generalization, i.e., the extent to which models trained on a set of (seen) visual concepts can be leveraged to recognize a new set of (unseen) concepts, is a popular way of evaluating visual representations, especially in a self-supervised learning framework. Nonetheless, the choice of unseen concepts for such an evaluation is usually made arbitrarily, and independently from the seen concepts used to train representations, thus ignoring any semantic relationships between the two. In this paper, we argue that the semantic relationships between seen and unseen concepts affect generalization performance and propose ImageNet-CoG, a novel benchmark on the ImageNet-21K (IN-21K) dataset that enables measuring concept generalization in a principled way. Our benchmark leverages expert knowledge that comes from WordNet in order to define a sequence of unseen IN-21K concept sets that are semantically more and more distant from the ImageNet-1K (IN-1K) subset, a ubiquitous training set. This allows us to benchmark visual representations learned on IN-1K out-of-the box. We conduct a large-scale study encompassing 31 convolution and transformer-based models and show how different architectures, levels of supervision, regularization techniques and use of web data impact the concept generalization performance.