# ICAR: Image-based Complementary Auto Reasoning

arXiv: https://arxiv.org/abs/2308.09119 · HTML: https://mlsanity.com/paper/2308.09119/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 32 of its 61 references that are in the map.
- First posted on arXiv 17 Aug 2023.

## Builds on (most cited first, 30 of 32)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Taming Transformers for High-Resolution Image Synthesis](https://mlsanity.com/paper/2012.09841/) (2020)
- [XLNet: Generalized Autoregressive Pretraining for Language Understanding](https://mlsanity.com/paper/1906.08237/) (2019)
- [BEiT: BERT Pre-Training of Image Transformers](https://mlsanity.com/paper/2106.08254/) (2021)
- [Swin Transformer V2: Scaling Up Capacity and Resolution](https://mlsanity.com/paper/2111.09883/) (2021)
- [In Defense of the Triplet Loss for Person Re-Identification](https://mlsanity.com/paper/1703.07737/) (2017)
- [Image-based Recommendations on Styles and Substitutes](https://mlsanity.com/paper/1506.04757/) (2015)
- [Fine-tuning CNN Image Retrieval with No Human Annotation](https://mlsanity.com/paper/1711.02512/) (2017)
- [3D-FRONT: 3D Furnished Rooms with layOuts and semaNTics](https://mlsanity.com/paper/2011.09127/) (2020)
- [Video Action Transformer Network](https://mlsanity.com/paper/1812.02707/) (2018)
- [ProxyNCA++: Revisiting and Revitalizing Proxy Neighborhood Component Analysis](https://mlsanity.com/paper/2004.01113/) (2020)
- [Training Vision Transformers for Image Retrieval](https://mlsanity.com/paper/2102.05644/) (2021)
- [Learning Fashion Compatibility with Bidirectional LSTMs](https://mlsanity.com/paper/1707.05691/) (2017)
- [POG: Personalized Outfit Generation for Fashion Recommendation at Alibaba iFashion](https://mlsanity.com/paper/1905.01866/) (2019)
- [Learning Type-Aware Embeddings for Fashion Compatibility](https://mlsanity.com/paper/1803.09196/) (2018)
- [Cross-domain Image Retrieval with a Dual Attribute-aware Ranking Network](https://mlsanity.com/paper/1505.07922/) (2015)
- [Fashion Outfit Complementary Item Retrieval](https://mlsanity.com/paper/1912.08967/) (2019)
- [Learning Similarity Conditions Without Explicit Supervision](https://mlsanity.com/paper/1908.08589/) (2019)
- [Learning Visual Clothing Style with Heterogeneous Dyadic Co-occurrences](https://mlsanity.com/paper/1509.07473/) (2015)
- [Hierarchical Fashion Graph Network for Personalized Outfit Recommendation](https://mlsanity.com/paper/2005.12566/) (2020)
- [Video2Shop: Exact Matching Clothes in Videos to Online Shopping Images](https://mlsanity.com/paper/1804.05287/) (2018)
- [Creating Capsule Wardrobes from Fashion Images](https://mlsanity.com/paper/1712.02662/) (2017)
- [Complete the Look: Scene-based Complementary Product Recommendation](https://mlsanity.com/paper/1812.01748/) (2018)
- [Combination of Multiple Global Descriptors for Image Retrieval](https://mlsanity.com/paper/1903.10663/) (2019)
- [Learning the Latent "Look": Unsupervised Discovery of a Style-Coherent Embedding from Fashion Images](https://mlsanity.com/paper/1707.03376/) (2017)
- [Mining Fashion Outfit Composition Using An End-to-End Deep Learning Approach on Set Data](https://mlsanity.com/paper/1608.03016/) (2016)
- [Visual Discovery at Pinterest](https://mlsanity.com/paper/1702.04680/) (2017)
- [Compatibility Family Learning for Item Recommendation and Generation](https://mlsanity.com/paper/1712.01262/) (2017)
- [Recommending Outfits from Personal Closet](https://mlsanity.com/paper/1804.09979/) (2018)

## Abstract (arXiv)

Scene-aware Complementary Item Retrieval (CIR) is a challenging task which requires to generate a set of compatible items across domains. Due to the subjectivity, it is difficult to set up a rigorous standard for both data collection and learning objectives. To address this challenging task, we propose a visual compatibility concept, composed of similarity (resembling in color, geometry, texture, and etc.) and complementarity (different items like table vs chair completing a group). Based on this notion, we propose a compatibility learning framework, a category-aware Flexible Bidirectional Transformer (FBT), for visual "scene-based set compatibility reasoning" with the cross-domain visual similarity input and auto-regressive complementary item generation. We introduce a "Flexible Bidirectional Transformer (FBT)" consisting of an encoder with flexible masking, a category prediction arm, and an auto-regressive visual embedding prediction arm. And the inputs for FBT are cross-domain visual similarity invariant embeddings, making this framework quite generalizable. Furthermore, our proposed FBT model learns the inter-object compatibility from a large set of scene images in a self-supervised way. Compared with the SOTA methods, this approach achieves up to 5.3% and 9.6% in FITB score and 22.3% and 31.8% SFID improvement on fashion and furniture, respectively.