# FashionM3: Multimodal, Multitask, and Multiround Fashion Assistant based on Unified Vision-Language Model

arXiv: https://arxiv.org/abs/2504.17826 · HTML: https://mlsanity.com/paper/2504.17826/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Placed in Document understanding and multimodal RAG, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 27 of its 66 references that are in the map.
- First posted on arXiv 24 Apr 2025.

## Placed in

- Multimodal LLMs / vision-language models / [Document understanding and multimodal RAG](https://mlsanity.com/mllm/document-understanding-and-multimodal-rag/)

## Builds on (most cited first, 27 of 27)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [Diffusion Models Beat GANs on Image Synthesis](https://mlsanity.com/paper/2105.05233/) (2021)
- [GPT-4o System Card](https://mlsanity.com/paper/2410.21276/) (2024)
- [Show-o: One Single Transformer to Unify Multimodal Understanding and Generation](https://mlsanity.com/paper/2408.12528/) (2024)
- [Image-based Recommendations on Styles and Substitutes](https://mlsanity.com/paper/1506.04757/) (2015)
- [Language Model Beats Diffusion -- Tokenizer is Key to Visual Generation](https://mlsanity.com/paper/2310.05737/) (2023)
- [Textbooks Are All You Need](https://mlsanity.com/paper/2306.11644/) (2023)
- [JanusFlow: Harmonizing Autoregression and Rectified Flow for Unified Multimodal Understanding and Generation](https://mlsanity.com/paper/2411.07975/) (2024)
- [MEDEC: A Benchmark for Medical Error Detection and Correction in Clinical Notes](https://mlsanity.com/paper/2412.19260/) (2024)
- [Generative Recommendation: Towards Next-generation Recommender Paradigm](https://mlsanity.com/paper/2304.03516/) (2023)
- [POG: Personalized Outfit Generation for Fashion Recommendation at Alibaba iFashion](https://mlsanity.com/paper/1905.01866/) (2019)
- [Learning Type-Aware Embeddings for Fashion Compatibility](https://mlsanity.com/paper/1803.09196/) (2018)
- [Vitron: A Unified Pixel-level Vision LLM for Understanding, Generating, Segmenting, Editing](https://mlsanity.com/paper/2412.19806/) (2024)
- [Fashion Outfit Complementary Item Retrieval](https://mlsanity.com/paper/1912.08967/) (2019)
- [Learning Similarity Conditions Without Explicit Supervision](https://mlsanity.com/paper/1908.08589/) (2019)
- [Diffusion Models for Generative Outfit Recommendation](https://mlsanity.com/paper/2402.17279/) (2024)
- [Hierarchical Fashion Graph Network for Personalized Outfit Recommendation](https://mlsanity.com/paper/2005.12566/) (2020)
- [A Review of Modern Fashion Recommender Systems](https://mlsanity.com/paper/2202.02757/) (2022)
- [UniFashion: A Unified Vision-Language Model for Multimodal Fashion Retrieval and Generation](https://mlsanity.com/paper/2408.11305/) (2024)
- [Explainable Outfit Recommendation with Joint Outfit Matching and Comment Generation](https://mlsanity.com/paper/1806.08977/) (2018)
- [Leveraging Multimodal Features and Item-level User Feedback for Bundle Construction](https://mlsanity.com/paper/2310.18770/) (2023)
- [Toward Explainable Fashion Recommendation](https://mlsanity.com/paper/1901.04870/) (2019)
- [Fashion Outfit Generation for E-commerce](https://mlsanity.com/paper/1904.00741/) (2019)
- [Theme-Matters: Fashion Compatibility Learning via Theme Attention](https://mlsanity.com/paper/1912.06227/) (2019)
- [Integrating Domain Knowledge into Large Language Models for Enhanced Fashion Recommendations](https://mlsanity.com/paper/2502.15696/) (2025)

## Built on by (most cited first, 2 of 2)

- [Discrete Tokenization for Multimodal LLMs: A Comprehensive Survey](https://mlsanity.com/paper/2507.22920/) (2025)
- [OmniFashion: Towards Generalist Fashion Intelligence via Multi-Task Vision-Language Learning](https://mlsanity.com/paper/2603.02658/) (2026)

## Abstract (arXiv)

Fashion styling and personalized recommendations are pivotal in modern retail, contributing substantial economic value in the fashion industry. With the advent of vision-language models (VLM), new opportunities have emerged to enhance retailing through natural language and visual interactions. This work proposes FashionM3, a multimodal, multitask, and multiround fashion assistant, built upon a VLM fine-tuned for fashion-specific tasks. It helps users discover satisfying outfits by offering multiple capabilities including personalized recommendation, alternative suggestion, product image generation, and virtual try-on simulation. Fine-tuned on the novel FashionRec dataset, comprising 331,124 multimodal dialogue samples across basic, personalized, and alternative recommendation tasks, FashionM3 delivers contextually personalized suggestions with iterative refinement through multiround interactions. Quantitative and qualitative evaluations, alongside user studies, demonstrate FashionM3's superior performance in recommendation effectiveness and practical value as a fashion assistant.