# ARMANI: Part-level Garment-Text Alignment for Unified Cross-Modal Fashion Design

arXiv: https://arxiv.org/abs/2208.05621 · HTML: https://mlsanity.com/paper/2208.05621/

## Key facts

- Cited by 16 papers in the map (data through 14 Oct 2026).
- Builds on 34 of its 49 references that are in the map.
- First posted on arXiv 11 Aug 2022.

## Builds on (most cited first, 30 of 34)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [A Style-Based Generator Architecture for Generative Adversarial Networks](https://mlsanity.com/paper/1812.04948/) (2018)
- [Neural Discrete Representation Learning](https://mlsanity.com/paper/1711.00937/) (2017)
- [Zero-Shot Text-to-Image Generation](https://mlsanity.com/paper/2102.12092/) (2021)
- [Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision](https://mlsanity.com/paper/2102.05918/) (2021)
- [Taming Transformers for High-Resolution Image Synthesis](https://mlsanity.com/paper/2012.09841/) (2020)
- [Improved Techniques for Training GANs](https://mlsanity.com/paper/1606.03498/) (2016)
- [CLIPScore: A Reference-free Evaluation Metric for Image Captioning](https://mlsanity.com/paper/2104.08718/) (2021)
- [ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Tasks](https://mlsanity.com/paper/1908.02265/) (2019)
- [Generating Diverse High-Fidelity Images with VQ-VAE-2](https://mlsanity.com/paper/1906.00446/) (2019)
- [VisualBERT: A Simple and Performant Baseline for Vision and Language](https://mlsanity.com/paper/1908.03557/) (2019)
- [ViLT: Vision-and-Language Transformer Without Convolution or Region Supervision](https://mlsanity.com/paper/2102.03334/) (2021)
- [Grounded Language-Image Pre-training](https://mlsanity.com/paper/2112.03857/) (2021)
- [Transformers in Vision: A Survey](https://mlsanity.com/paper/2101.01169/) (2021)
- [StyleCLIP: Text-Driven Manipulation of StyleGAN Imagery](https://mlsanity.com/paper/2103.17249/) (2021)
- [Generative Adversarial Text to Image Synthesis](https://mlsanity.com/paper/1605.05396/) (2016)
- [CogView: Mastering Text-to-Image Generation via Transformers](https://mlsanity.com/paper/2105.13290/) (2021)
- [AttnGAN: Fine-Grained Text to Image Generation with Attentional Generative Adversarial Networks](https://mlsanity.com/paper/1711.10485/) (2017)
- [FILIP: Fine-grained Interactive Language-Image Pre-Training](https://mlsanity.com/paper/2111.07783/) (2021)
- [Cross-Modal Contrastive Learning for Text-to-Image Generation](https://mlsanity.com/paper/2101.04702/) (2021)
- [UNIMO: Towards Unified-Modal Understanding and Generation via Cross-Modal Contrastive Learning](https://mlsanity.com/paper/2012.15409/) (2020)
- [StackGAN++: Realistic Image Synthesis with Stacked Generative Adversarial Networks](https://mlsanity.com/paper/1710.10916/) (2017)
- [ManiGAN: Text-Guided Image Manipulation](https://mlsanity.com/paper/1912.06203/) (2019)
- [Object-driven Text-to-Image Synthesis via Adversarial Training](https://mlsanity.com/paper/1902.10740/) (2019)
- [Inferring Semantic Layout for Hierarchical Text-to-Image Synthesis](https://mlsanity.com/paper/1801.05091/) (2018)
- [Semantic Object Accuracy for Generative Text-to-Image Synthesis](https://mlsanity.com/paper/1910.13321/) (2019)
- [Connecting Touch and Vision via Cross-Modal Prediction](https://mlsanity.com/paper/1906.06322/) (2019)
- [Photographic Text-to-Image Synthesis with a Hierarchically-nested Adversarial Network](https://mlsanity.com/paper/1802.09178/) (2018)
- [An Introduction to Image Synthesis with Generative Adversarial Nets](https://mlsanity.com/paper/1803.04469/) (2018)
- [Cycle-Consistent Inverse GAN for Text-to-Image Synthesis](https://mlsanity.com/paper/2108.01361/) (2021)

## Built on by (most cited first, 16 of 16)

- [SGDiff: A Style Guided Diffusion Model for Fashion Synthesis](https://mlsanity.com/paper/2308.07605/) (2023)
- [Tell2Design: A Dataset for Language-Guided Floor Plan Generation](https://mlsanity.com/paper/2311.15941/) (2023)
- [TryOffDiff: Virtual-Try-Off via High-Fidelity Garment Reconstruction using Diffusion Models](https://mlsanity.com/paper/2411.18350/) (2024)
- [DiffCloth: Diffusion Based Garment Synthesis and Manipulation via Structural Cross-modal Semantic Alignment](https://mlsanity.com/paper/2308.11206/) (2023)
- [ViLLA: Fine-Grained Vision-Language Representation Learning from Real-World Data](https://mlsanity.com/paper/2308.11194/) (2023)
- [RAGDiffusion: Faithful Cloth Generation via External Knowledge Assimilation](https://mlsanity.com/paper/2411.19528/) (2024)
- [HieraFashDiff: Hierarchical Fashion Design with Multi-stage Diffusion Models](https://mlsanity.com/paper/2401.07450/) (2024)
- [MGT: Extending Virtual Try-Off to Multi-Garment Scenarios](https://mlsanity.com/paper/2504.13078/) (2025)
- [FIRST: A Million-Entry Dataset for Text-Driven Fashion Synthesis and Design](https://mlsanity.com/paper/2311.07414/) (2023)
- [Quality and Quantity: Unveiling a Million High-Quality Images for Text-to-Image Synthesis in Fashion Design](https://mlsanity.com/paper/2311.12067/) (2023)
- [HiGarment: Cross-modal Harmony Based Diffusion Model for Flat Sketch to Realistic Garment Image](https://mlsanity.com/paper/2505.23186/) (2025)
- [UniDiff: Advancing Vision-Language Models with Generative and Discriminative Learning](https://mlsanity.com/paper/2306.00813/) (2023)
- [EditGarment: An Instruction-Based Garment Editing Dataset Constructed with Automated MLLM Synthesis and Semantic-Aware Evaluation](https://mlsanity.com/paper/2508.03497/) (2025)
- [OmniDiT: Extending Diffusion Transformer to Omni-VTON Framework](https://mlsanity.com/paper/2603.19643/) (2026)
- [Bridging Text and Image for Artist Style Transfer via Contrastive Learning](https://mlsanity.com/paper/2410.09566/) (2024)
- [OmniFashion: Towards Generalist Fashion Intelligence via Multi-Task Vision-Language Learning](https://mlsanity.com/paper/2603.02658/) (2026)

## Abstract (arXiv)

Cross-modal fashion image synthesis has emerged as one of the most promising directions in the generation domain due to the vast untapped potential of incorporating multiple modalities and the wide range of fashion image applications. To facilitate accurate generation, cross-modal synthesis methods typically rely on Contrastive Language-Image Pre-training (CLIP) to align textual and garment information. In this work, we argue that simply aligning texture and garment information is not sufficient to capture the semantics of the visual information and therefore propose MaskCLIP. MaskCLIP decomposes the garments into semantic parts, ensuring fine-grained and semantically accurate alignment between the visual and text information. Building on MaskCLIP, we propose ARMANI, a unified cross-modal fashion designer with part-level garment-text alignment. ARMANI discretizes an image into uniform tokens based on a learned cross-modal codebook in its first stage and uses a Transformer to model the distribution of image tokens for a real image given the tokens of the control signals in its second stage. Contrary to prior approaches that also rely on two-stage paradigms, ARMANI introduces textual tokens into the codebook, making it possible for the model to utilize fine-grain semantic information to generate more realistic images. Further, by introducing a cross-modal Transformer, ARMANI is versatile and can accomplish image synthesis from various control signals, such as pure text, sketch images, and partial images. Extensive experiments conducted on our newly collected cross-modal fashion dataset demonstrate that ARMANI generates photo-realistic images in diverse synthesis tasks and outperforms existing state-of-the-art cross-modal image synthesis approaches.Our code is available at https://github.com/Harvey594/ARMANI.