# Fashion Captioning: Towards Generating Accurate Descriptions with Semantic Rewards

arXiv: https://arxiv.org/abs/2008.02693 · HTML: https://mlsanity.com/paper/2008.02693/

## Key facts

- Cited by 38 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 6 Aug 2020.

## Built on by (most cited first, 30 of 38)

- [From Show to Tell: A Survey on Deep Learning-based Image Captioning](https://mlsanity.com/paper/2107.06912/) (2021)
- [Kaleido-BERT: Vision-Language Pre-training on Fashion Domain](https://mlsanity.com/paper/2103.16110/) (2021)
- [Composed Image Retrieval with Text Feedback via Multi-grained Uncertainty Regularization](https://mlsanity.com/paper/2211.07394/) (2022)
- [FAME-ViL: Multi-Tasking Vision-Language Model for Heterogeneous Fashion Tasks](https://mlsanity.com/paper/2303.02483/) (2023)
- [FashionViL: Fashion-Focused Vision-and-Language Representation Learning](https://mlsanity.com/paper/2207.08150/) (2022)
- [S-CLIP: Semi-supervised Vision-Language Learning using Few Specialist Captions](https://mlsanity.com/paper/2305.14095/) (2023)
- [Progressive Learning for Image Retrieval with Hybrid-Modality Queries](https://mlsanity.com/paper/2204.11212/) (2022)
- [ReFormer: The Relational Transformer for Image Captioning](https://mlsanity.com/paper/2107.14178/) (2021)
- [UniFashion: A Unified Vision-Language Model for Multimodal Fashion Retrieval and Generation](https://mlsanity.com/paper/2408.11305/) (2024)
- [Journalistic Guidelines Aware News Image Captioning](https://mlsanity.com/paper/2109.02865/) (2021)
- [Masked Vision-Language Transformer in Fashion](https://mlsanity.com/paper/2210.15110/) (2022)
- [FashionSAP: Symbols and Attributes Prompt for Fine-grained Fashion Vision-Language Pre-training](https://mlsanity.com/paper/2304.05051/) (2023)
- [CapOnImage: Context-driven Dense-Captioning on Image](https://mlsanity.com/paper/2204.12974/) (2022)
- [FashionNTM: Multi-turn Fashion Image Retrieval via Cascaded Memory](https://mlsanity.com/paper/2308.10170/) (2023)
- [Product-oriented Machine Translation with Cross-modal Cross-lingual Pre-training](https://mlsanity.com/paper/2108.11119/) (2021)
- [FaD-VLP: Fashion Vision-and-Language Pre-training towards Unified Retrieval and Captioning](https://mlsanity.com/paper/2210.15028/) (2022)
- [Beyond Triplet: Leveraging the Most Data for Multimodal Machine Translation](https://mlsanity.com/paper/2212.10313/) (2022)
- [MOON: Generative MLLM-based Multimodal Representation Learning for E-commerce Product Understanding](https://mlsanity.com/paper/2508.11999/) (2025)
- [Multimodal Conditionality for Natural Language Generation](https://mlsanity.com/paper/2109.01229/) (2021)
- [A Survey on Multi-modal Machine Translation: Tasks, Methods and Challenges](https://mlsanity.com/paper/2405.12669/) (2024)
- [Neural Fashion Image Captioning : Accounting for Data Diversity](https://mlsanity.com/paper/2106.12154/) (2021)
- [Smart Fashion: A Review of AI Applications in the Fashion & Apparel Industry](https://mlsanity.com/paper/2111.00905/) (2021)
- [Paraphrasing Is All You Need for Novel Object Captioning](https://mlsanity.com/paper/2209.12343/) (2022)
- [Seeing the Abstract: Translating the Abstract Language for Vision Language Models](https://mlsanity.com/paper/2505.03242/) (2025)
- [GLAMI-1M: A Multilingual Image-Text Fashion Dataset](https://mlsanity.com/paper/2211.14451/) (2022)
- [Composed Multi-modal Retrieval: A Survey of Approaches and Applications](https://mlsanity.com/paper/2503.01334/) (2025)
- [Surveying the Landscape of Image Captioning Evaluation: A Comprehensive Taxonomy, Trends and Metrics Analysis](https://mlsanity.com/paper/2408.04909/) (2024)
- [FACap: A Large-scale Fashion Dataset for Fine-grained Composed Image Retrieval](https://mlsanity.com/paper/2507.07135/) (2025)
- [XFBoost: Improving Text Generation with Controllable Decoders](https://mlsanity.com/paper/2202.08124/) (2022)
- [Controllable Image Captioning](https://mlsanity.com/paper/2204.13324/) (2022)

## Abstract (arXiv)

Generating accurate descriptions for online fashion items is important not only for enhancing customers' shopping experiences, but also for the increase of online sales. Besides the need of correctly presenting the attributes of items, the expressions in an enchanting style could better attract customer interests. The goal of this work is to develop a novel learning framework for accurate and expressive fashion captioning. Different from popular work on image captioning, it is hard to identify and describe the rich attributes of fashion items. We seed the description of an item by first identifying its attributes, and introduce attribute-level semantic (ALS) reward and sentence-level semantic (SLS) reward as metrics to improve the quality of text descriptions. We further integrate the training of our model with maximum likelihood estimation (MLE), attribute embedding, and Reinforcement Learning (RL). To facilitate the learning, we build a new FAshion CAptioning Dataset (FACAD), which contains 993K images and 130K corresponding enchanting and diverse descriptions. Experiments on FACAD demonstrate the effectiveness of our model.