# Fashion Matrix: Editing Photos by Just Talking

arXiv: https://arxiv.org/abs/2307.13240 · HTML: https://mlsanity.com/paper/2307.13240/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 32 of its 38 references that are in the map.
- First posted on arXiv 25 Jul 2023.

## Builds on (most cited first, 30 of 32)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [ReAct: Synergizing Reasoning and Acting in Language Models](https://mlsanity.com/paper/2210.03629/) (2022)
- [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://mlsanity.com/paper/2306.05685/) (2023)
- [Adding Conditional Control to Text-to-Image Diffusion Models](https://mlsanity.com/paper/2302.05543/) (2023)
- [PaLM: Scaling Language Modeling with Pathways](https://mlsanity.com/paper/2204.02311/) (2022)
- [BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation](https://mlsanity.com/paper/2201.12086/) (2022)
- [Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detection](https://mlsanity.com/paper/2303.05499/) (2023)
- [Improved Techniques for Training GANs](https://mlsanity.com/paper/1606.03498/) (2016)
- [HuggingGPT: Solving AI Tasks with ChatGPT and its Friends in Hugging Face](https://mlsanity.com/paper/2303.17580/) (2023)
- [OpenPose: Realtime Multi-Person 2D Pose Estimation using Part Affinity Fields](https://mlsanity.com/paper/1812.08008/) (2018)
- [GLM-130B: An Open Bilingual Pre-trained Model](https://mlsanity.com/paper/2210.02414/) (2022)
- [Automatic Chain of Thought Prompting in Large Language Models](https://mlsanity.com/paper/2210.03493/) (2022)
- [Visual ChatGPT: Talking, Drawing and Editing with Visual Foundation Models](https://mlsanity.com/paper/2303.04671/) (2023)
- [DensePose: Dense Human Pose Estimation In The Wild](https://mlsanity.com/paper/1802.00434/) (2018)
- [VITON-HD: High-Resolution Virtual Try-On via Misalignment-Aware Normalization](https://mlsanity.com/paper/2103.16874/) (2021)
- [CrowdPose: Efficient Crowded Scenes Pose Estimation and A New Benchmark](https://mlsanity.com/paper/1812.00324/) (2018)
- [AlphaPose: Whole-Body Regional Multi-Person Pose Estimation and Tracking in Real-Time](https://mlsanity.com/paper/2211.03375/) (2022)
- [Toward Characteristic-Preserving Image-based Virtual Try-On Network](https://mlsanity.com/paper/1807.07688/) (2018)
- [Pixel Difference Networks for Efficient Edge Detection](https://mlsanity.com/paper/2108.07009/) (2021)
- [ManiGAN: Text-Guided Image Manipulation](https://mlsanity.com/paper/1912.06203/) (2019)
- [StyleGAN-Human: A Data-Centric Odyssey of Human Generation](https://mlsanity.com/paper/2204.11823/) (2022)
- [TryOnDiffusion: A Tale of Two UNets](https://mlsanity.com/paper/2306.08276/) (2023)
- [Instance-level Human Parsing via Part Grouping Network](https://mlsanity.com/paper/1808.00157/) (2018)
- [Look into Person: Self-supervised Structure-sensitive Learning and A New Benchmark for Human Parsing](https://mlsanity.com/paper/1703.05446/) (2017)
- [Graphonomy: Universal Human Parsing via Graph Transfer Learning](https://mlsanity.com/paper/1904.04536/) (2019)
- [Look into Person: Joint Body Parsing & Pose Estimation Network and A New Benchmark](https://mlsanity.com/paper/1804.01984/) (2018)
- [FICE: Text-Conditioned Fashion Image Editing With Guided GAN Inversion](https://mlsanity.com/paper/2301.02110/) (2023)

## Built on by (most cited first, 1 of 1)

- [Total Selfie: Generating Full-Body Selfies](https://mlsanity.com/paper/2308.14740/) (2023)

## Abstract (arXiv)

The utilization of Large Language Models (LLMs) for the construction of AI systems has garnered significant attention across diverse fields. The extension of LLMs to the domain of fashion holds substantial commercial potential but also inherent challenges due to the intricate semantic interactions in fashion-related generation. To address this issue, we developed a hierarchical AI system called Fashion Matrix dedicated to editing photos by just talking. This system facilitates diverse prompt-driven tasks, encompassing garment or accessory replacement, recoloring, addition, and removal. Specifically, Fashion Matrix employs LLM as its foundational support and engages in iterative interactions with users. It employs a range of Semantic Segmentation Models (e.g., Grounded-SAM, MattingAnything, etc.) to delineate the specific editing masks based on user instructions. Subsequently, Visual Foundation Models (e.g., Stable Diffusion, ControlNet, etc.) are leveraged to generate edited images from text prompts and masks, thereby facilitating the automation of fashion editing processes. Experiments demonstrate the outstanding ability of Fashion Matrix to explores the collaborative potential of functionally diverse pre-trained models in the domain of fashion editing.