# The MODA General Attribute Suite: A Four-Track Evaluation Benchmark for Fashion Attribute Extraction

arXiv: https://arxiv.org/abs/2609.13279 · HTML: https://mlsanity.com/paper/2609.13279/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 10 of its 14 references that are in the map.
- First posted on arXiv 8 Sep 2026.

## Builds on (most cited first, 10 of 10)

- [Sigmoid Loss for Language Image Pre-Training](https://mlsanity.com/paper/2303.15343/) (2023)
- [Selective Classification for Deep Neural Networks](https://mlsanity.com/paper/1705.08500/) (2017)
- [Florence-2: Advancing a Unified Representation for a Variety of Vision Tasks](https://mlsanity.com/paper/2311.06242/) (2023)
- [Leakage and the Reproducibility Crisis in ML-based Science](https://mlsanity.com/paper/2207.07048/) (2022)
- [Fashionpedia: Ontology, Segmentation, and an Attribute Localization Dataset](https://mlsanity.com/paper/2004.12276/) (2020)
- [Cross-domain Image Retrieval with a Dual Attribute-aware Ranking Network](https://mlsanity.com/paper/1505.07922/) (2015)
- [Contrastive language and vision learning of general fashion concepts](https://mlsanity.com/paper/2204.03972/) (2022)
- [LookBench: A Live and Holistic Open Benchmark for Fashion Image Retrieval](https://mlsanity.com/paper/2601.14706/) (2026)
- [Bridging Annotation Gaps: Transferring Labels to Align Object Detection Datasets](https://mlsanity.com/paper/2506.04737/) (2025)
- [Zero-Shot Product Attribute Labeling with Vision-Language Models: A Three-Tier Evaluation Framework](https://mlsanity.com/paper/2601.15711/) (2026)

## Abstract (arXiv)

Fashion attribute extraction is evaluated inconsistently: results are reported as single aggregate numbers across image types that pose different problems, fields that are not visible in an image are scored as ordinary negatives, and the effect of vocabulary mismatch between datasets is acknowledged but not measured. We release the MODA General Attribute Suite, a four-track benchmark that keeps these problems separate by construction. Each track (localized garment crops, catalogue product images, applicability-aware full-body photographs, and product text) carries its own frozen test set, input contract, metric, and leakage unit, and the tracks are never averaged. The protocol requires label-blind prediction, SHA-256 commitment of prediction files before any label is opened, a fail-closed scorer, and a 10,000-sample paired cluster bootstrap at each track's natural leakage unit; promotion requires a positive interval on every track rather than a favourable mean. We release the scorers, the split builders, our own prediction files with their hashes for every image-track system in the main results and for runs we lose, and the MODA_NER(V) model checkpoints for three of the four tracks. The text model is not distributed; its benchmark is. We report baseline results, six interventions that did not improve them, and the limitations of the evidence. On the garment-crop track we also measure how far the headline depends on the benchmark's own design: the order of the two strongest systems depends on whether fields are micro- or macro-averaged, two thirds of the cells the protocol judges carry no annotation, and disabling the applicability decision cuts attribute micro-F1 from 0.63 to 0.27, so most of that score rests on absence decisions the source labels cannot adjudicate. This version also corrects one figure in an external comparison that an inference preprocessing defect had affected.