# Elucidating image-to-set prediction: An analysis of models, losses and datasets

arXiv: https://arxiv.org/abs/1904.05709 · HTML: https://mlsanity.com/paper/1904.05709/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 11 Apr 2019.

## Built on by (most cited first, 10 of 10)

- [Multi-Label Learning from Single Positive Labels](https://mlsanity.com/paper/2106.09708/) (2021)
- [Simple and Robust Loss Design for Multi-Label Learning with Missing Labels](https://mlsanity.com/paper/2112.07368/) (2021)
- [Learning to Substitute Ingredients in Recipes](https://mlsanity.com/paper/2302.07960/) (2023)
- [G2NetPL: Generic Game-Theoretic Network for Partial-Label Image Classification](https://mlsanity.com/paper/2210.11469/) (2022)
- [PLMCL: Partial-Label Momentum Curriculum Learning for Multi-Label Image Classification](https://mlsanity.com/paper/2208.09999/) (2022)
- [DATE: Dual Assignment for End-to-End Fully Convolutional Object Detection](https://mlsanity.com/paper/2211.13859/) (2022)
- [Neural Slot Interpreters: Grounding Object Semantics in Emergent Slot Representations](https://mlsanity.com/paper/2403.07887/) (2024)
- [Goal-driven text descriptions for images](https://mlsanity.com/paper/2108.12575/) (2021)
- [Improving Token-based Object Detection with Video](https://mlsanity.com/paper/2506.22562/) (2025)
- [Learning Attribute-Aware Hash Codes for Fine-Grained Image Retrieval via Query Optimization](https://mlsanity.com/paper/2509.17049/) (2025)

## Abstract (arXiv)

In this paper, we identify an important reproducibility challenge in the image-to-set prediction literature that impedes proper comparisons among published methods, namely, researchers use different evaluation protocols to assess their contributions. To alleviate this issue, we introduce an image-to-set prediction benchmark suite built on top of five public datasets of increasing task complexity that are suitable for multi-label classification (VOC, COCO, NUS-WIDE, ADE20k and Recipe1M). Using the benchmark, we provide an in-depth analysis where we study the key components of current models, namely the choice of the image representation backbone as well as the set predictor design. Our results show that (1) exploiting better image representation backbones leads to higher performance boosts than enhancing set predictors, and (2) modeling both the label co-occurrences and ordering has a slight positive impact in terms of performance, whereas explicit cardinality prediction only helps when training on complex datasets, such as Recipe1M. To facilitate future image-to-set prediction research, we make the code, best models and dataset splits publicly available at: https://github.com/facebookresearch/image-to-set.