# Unlocking ImageNet's Multi-Object Nature: Automated Large-Scale Multilabel Annotation

arXiv: https://arxiv.org/abs/2603.05729 · HTML: https://mlsanity.com/paper/2603.05729/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 30 of its 37 references that are in the map.
- First posted on arXiv 5 Mar 2026.

## Builds on (most cited first, 30 of 30)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [ImageNet Large Scale Visual Recognition Challenge](https://mlsanity.com/paper/1409.0575/) (2014)
- [DINOv2: Learning Robust Visual Features without Supervision](https://mlsanity.com/paper/2304.07193/) (2023)
- [Emerging Properties in Self-Supervised Vision Transformers](https://mlsanity.com/paper/2104.14294/) (2021)
- [Flamingo: a Visual Language Model for Few-Shot Learning](https://mlsanity.com/paper/2204.14198/) (2022)
- [BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation](https://mlsanity.com/paper/2201.12086/) (2022)
- [SAM 2: Segment Anything in Images and Videos](https://mlsanity.com/paper/2408.00714/) (2024)
- [RandAugment: Practical automated data augmentation with a reduced search space](https://mlsanity.com/paper/1909.13719/) (2019)
- [DINOv3](https://mlsanity.com/paper/2508.10104/) (2025)
- [Cascade R-CNN: Delving into High Quality Object Detection](https://mlsanity.com/paper/1712.00726/) (2017)
- [Class-Balanced Loss Based on Effective Number of Samples](https://mlsanity.com/paper/1901.05555/) (2019)
- [ImageNet-21K Pretraining for the Masses](https://mlsanity.com/paper/2104.10972/) (2021)
- [DeiT III: Revenge of the ViT](https://mlsanity.com/paper/2204.07118/) (2022)
- [Asymmetric Loss For Multi-Label Classification](https://mlsanity.com/paper/2009.14119/) (2020)
- [EVA-02: A Visual Representation for Neon Genesis](https://mlsanity.com/paper/2303.11331/) (2023)
- [ResNet strikes back: An improved training procedure in timm](https://mlsanity.com/paper/2110.00476/) (2021)
- [Are we done with ImageNet?](https://mlsanity.com/paper/2006.07159/) (2020)
- [Cut and Learn for Unsupervised Object Detection and Instance Segmentation](https://mlsanity.com/paper/2301.11320/) (2023)
- [Query2Label: A Simple Transformer Way to Multi-Label Classification](https://mlsanity.com/paper/2107.10834/) (2021)
- [Re-labeling ImageNet: from Single to Multi-Labels, from Global to Localized Labels](https://mlsanity.com/paper/2101.05022/) (2021)
- [TokenCut: Segmenting Objects in Images and Videos with Self-supervised Transformer and Normalized Cut](https://mlsanity.com/paper/2209.00383/) (2022)
- [Can multi-label classification networks know what they don't know?](https://mlsanity.com/paper/2109.14162/) (2021)
- [When does dough become a bagel? Analyzing the remaining mistakes on ImageNet](https://mlsanity.com/paper/2205.04596/) (2022)
- [Large Loss Matters in Weakly Supervised Multi-Label Classification](https://mlsanity.com/paper/2206.03740/) (2022)
- [Spatial Consistency Loss for Training Multi-Label Classifiers from Single-Label Annotations](https://mlsanity.com/paper/2203.06127/) (2022)
- [What Variables Affect Out-of-Distribution Generalization in Pretrained Models?](https://mlsanity.com/paper/2405.15018/) (2024)
- [Controlling Neural Collapse Enhances Out-of-Distribution Detection and Transfer Learning](https://mlsanity.com/paper/2502.10691/) (2025)
- [Leveraging Human-Machine Interactions for Computer Vision Dataset Quality Enhancement](https://mlsanity.com/paper/2401.17736/) (2024)

## Abstract (arXiv)

The original ImageNet benchmark enforces a single-label assumption, despite many images depicting multiple objects. This leads to label noise and limits the richness of the learning signal. Multi-label annotations more accurately reflect real-world visual scenes, where multiple objects co-occur and contribute to semantic understanding, enabling models to learn richer and more robust representations. While prior efforts (e.g., ReaL, ImageNetv2) have improved the validation set, there has not yet been a scalable, high-quality multi-label annotation for the training set. To this end, we present an automated pipeline to convert the ImageNet training set into a multi-label dataset, without human annotations. Using self-supervised Vision Transformers, we perform unsupervised object discovery, select regions aligned with original labels to train a lightweight classifier, and apply it to all regions to generate coherent multi-label annotations across the dataset. Our labels show strong alignment with human judgment in qualitative evaluations and consistently improve performance across quantitative benchmarks. Compared to traditional single-label scheme, models trained with our multi-label supervision achieve consistently better in-domain accuracy across architectures (up to +2.0 top-1 accuracy on ReaL and +1.5 on ImageNet-V2) and exhibit stronger transferability to downstream tasks (up to +4.2 and +2.3 mAP on COCO and VOC, respectively). These results underscore the importance of accurate multi-label annotations for enhancing both classification performance and representation learning. Project code and the generated multi-label annotations are available at https://github.com/jchen175/MultiLabel-ImageNet.