# OutfitTransformer: Learning Outfit Representations for Fashion Recommendation

arXiv: https://arxiv.org/abs/2204.04812 · HTML: https://mlsanity.com/paper/2204.04812/

## Key facts

- Cited by 24 papers in the map (data through 14 Oct 2026).
- Builds on 27 of its 35 references that are in the map.
- First posted on arXiv 11 Apr 2022.

## Builds on (most cited first, 27 of 27)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [End-to-End Object Detection with Transformers](https://mlsanity.com/paper/2005.12872/) (2020)
- [Training data-efficient image transformers & distillation through attention](https://mlsanity.com/paper/2012.12877/) (2020)
- [SegFormer: Simple and Efficient Design for Semantic Segmentation with Transformers](https://mlsanity.com/paper/2105.15203/) (2021)
- [FaceNet: A Unified Embedding for Face Recognition and Clustering](https://mlsanity.com/paper/1503.03832/) (2015)
- [CrossViT: Cross-Attention Multi-Scale Vision Transformer for Image Classification](https://mlsanity.com/paper/2103.14899/) (2021)
- [Focal Loss for Dense Object Detection](https://mlsanity.com/paper/1708.02002/) (2017)
- [Stand-Alone Self-Attention in Vision Models](https://mlsanity.com/paper/1906.05909/) (2019)
- [Deep Metric Learning via Lifted Structured Feature Embedding](https://mlsanity.com/paper/1511.06452/) (2015)
- [Learning Texture Transformer Network for Image Super-Resolution](https://mlsanity.com/paper/2006.04139/) (2020)
- [Focal Self-attention for Local-Global Interactions in Vision Transformers](https://mlsanity.com/paper/2107.00641/) (2021)
- [Sampling Matters in Deep Embedding Learning](https://mlsanity.com/paper/1706.07567/) (2017)
- [Hard negative examples are hard, but useful](https://mlsanity.com/paper/2007.12749/) (2020)
- [Learning Fashion Compatibility with Bidirectional LSTMs](https://mlsanity.com/paper/1707.05691/) (2017)
- [POG: Personalized Outfit Generation for Fashion Recommendation at Alibaba iFashion](https://mlsanity.com/paper/1905.01866/) (2019)
- [Learning Type-Aware Embeddings for Fashion Compatibility](https://mlsanity.com/paper/1803.09196/) (2018)
- [Fashion Outfit Complementary Item Retrieval](https://mlsanity.com/paper/1912.08967/) (2019)
- [Learning Similarity Conditions Without Explicit Supervision](https://mlsanity.com/paper/1908.08589/) (2019)
- [Learning Visual Clothing Style with Heterogeneous Dyadic Co-occurrences](https://mlsanity.com/paper/1509.07473/) (2015)
- [Context-Aware Visual Compatibility Prediction](https://mlsanity.com/paper/1902.03646/) (2019)
- [Dressing as a Whole: Outfit Compatibility Learning Based on Node-wise Graph Neural Networks](https://mlsanity.com/paper/1902.08009/) (2019)
- [Creating Capsule Wardrobes from Fashion Images](https://mlsanity.com/paper/1712.02662/) (2017)
- [Learning the Latent "Look": Unsupervised Discovery of a Style-Coherent Embedding from Fashion Images](https://mlsanity.com/paper/1707.03376/) (2017)
- [Coherent and Controllable Outfit Generation](https://mlsanity.com/paper/1906.07273/) (2019)
- [TransNFCM: Translation-Based Neural Fashion Compatibility Modeling](https://mlsanity.com/paper/1812.10021/) (2018)

## Built on by (most cited first, 24 of 24)

- [LaDI-VTON: Latent Diffusion Textual-Inversion Enhanced Virtual Try-On](https://mlsanity.com/paper/2305.13501/) (2023)
- [Multimodal Garment Designer: Human-Centric Latent Diffusion Models for Fashion Image Editing](https://mlsanity.com/paper/2304.02051/) (2023)
- [A Survey of Foundation Model-Powered Recommender Systems: From Feature-Based, Generative to Agentic Paradigms](https://mlsanity.com/paper/2504.16420/) (2025)
- [ITA-MDT: Image-Timestep-Adaptive Masked Diffusion Transformer Framework for Image-Based Virtual Try-On](https://mlsanity.com/paper/2503.20418/) (2025)
- [OpenFashionCLIP: Vision-and-Language Contrastive Learning with Open-Source Fashion Data](https://mlsanity.com/paper/2309.05551/) (2023)
- [Multimodal-Conditioned Latent Diffusion Models for Fashion Image Editing](https://mlsanity.com/paper/2403.14828/) (2024)
- [Deciphering Compatibility Relationships with Textual Descriptions via Extraction and Explanation](https://mlsanity.com/paper/2312.11554/) (2023)
- [Survey for Landing Generative AI in Social and E-commerce Recsys -- the Industry Perspectives](https://mlsanity.com/paper/2406.06475/) (2024)
- [Dual Pose-invariant Embeddings: Learning Category and Object-specific Discriminative Representations for Recognition and Retrieval](https://mlsanity.com/paper/2403.00272/) (2024)
- [Decoding Style: Efficient Fine-Tuning of LLMs for Image-Guided Outfit Recommendation with Preference](https://mlsanity.com/paper/2409.12150/) (2024)
- [Hybrid-Hierarchical Fashion Graph Attention Network for Compatibility-Oriented and Personalized Outfit Recommendation](https://mlsanity.com/paper/2508.11105/) (2025)
- [TATTOO: Training-free AesTheTic-aware Outfit recOmmendation](https://mlsanity.com/paper/2509.23242/) (2025)
- [Lost Your Style? Navigating with Semantic-Level Approach for Text-to-Outfit Retrieval](https://mlsanity.com/paper/2311.02122/) (2023)
- [Character-based Outfit Generation with Vision-augmented Style Extraction via LLMs](https://mlsanity.com/paper/2402.05941/) (2024)
- [A Dataset and Framework for Learning State-invariant Object Representations](https://mlsanity.com/paper/2404.06470/) (2024)
- [FashionFail: Addressing Failure Cases in Fashion Object Detection and Segmentation](https://mlsanity.com/paper/2404.08582/) (2024)
- [Integrating Domain Knowledge into Large Language Models for Enhanced Fashion Recommendations](https://mlsanity.com/paper/2502.15696/) (2025)
- [Content-based jewellery item retrieval using the local region-based histograms](https://mlsanity.com/paper/2305.07540/) (2023)
- [DPDEdit: Detail-Preserved Diffusion Models for Multimodal Fashion Image Editing](https://mlsanity.com/paper/2409.01086/) (2024)
- [A Hybrid Multimodal Deep Learning Framework for Intelligent Fashion Recommendation](https://mlsanity.com/paper/2511.07573/) (2025)
- [TrendGen: An Outfit Recommendation and Display System](https://mlsanity.com/paper/2603.27264/) (2026)
- [Loom: Hybrid Retrieval-Scoring Outfit Recommendation with Semantic Material Compatibility and Occasion-Aware Embedding Priors](https://mlsanity.com/paper/2605.09830/) (2026)
- [FDM-MFVT: Few-step Sampling Diffusion Model for Mask-Free Virtual Try-On](https://mlsanity.com/paper/2606.29319/) (2026)
- [Autonomous Fashion Outfit Composition via Unified Aesthetic Foresight Model](https://mlsanity.com/paper/2608.13888/) (2026)

## Abstract (arXiv)

Learning an effective outfit-level representation is critical for predicting the compatibility of items in an outfit, and retrieving complementary items for a partial outfit. We present a framework, OutfitTransformer, that uses the proposed task-specific tokens and leverages the self-attention mechanism to learn effective outfit-level representations encoding the compatibility relationships between all items in the entire outfit for addressing both compatibility prediction and complementary item retrieval tasks. For compatibility prediction, we design an outfit token to capture a global outfit representation and train the framework using a classification loss. For complementary item retrieval, we design a target item token that additionally takes the target item specification (in the form of a category or text description) into consideration. We train our framework using a proposed set-wise outfit ranking loss to generate a target item embedding given an outfit, and a target item specification as inputs. The generated target item embedding is then used to retrieve compatible items that match the rest of the outfit. Additionally, we adopt a pre-training approach and a curriculum learning strategy to improve retrieval performance. Since our framework learns at an outfit-level, it allows us to learn a single embedding capturing higher-order relations among multiple items in the outfit more effectively than pairwise methods. Experiments demonstrate that our approach outperforms state-of-the-art methods on compatibility prediction, fill-in-the-blank, and complementary item retrieval tasks. We further validate the quality of our retrieval results with a user study.