# FashionVQA: A Domain-Specific Visual Question Answering System

arXiv: https://arxiv.org/abs/2208.11253 · HTML: https://mlsanity.com/paper/2208.11253/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 28 of its 43 references that are in the map.
- First posted on arXiv 24 Aug 2022.

## Builds on (most cited first, 28 of 28)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks](https://mlsanity.com/paper/1506.01497/) (2015)
- [ImageNet Large Scale Visual Recognition Challenge](https://mlsanity.com/paper/1409.0575/) (2014)
- [Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms](https://mlsanity.com/paper/1708.07747/) (2017)
- [Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation](https://mlsanity.com/paper/1406.1078/) (2014)
- [Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotations](https://mlsanity.com/paper/1602.07332/) (2016)
- [Aggregated Residual Transformations for Deep Neural Networks](https://mlsanity.com/paper/1611.05431/) (2016)
- [Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answering](https://mlsanity.com/paper/1612.00837/) (2016)
- [ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Tasks](https://mlsanity.com/paper/1908.02265/) (2019)
- [LXMERT: Learning Cross-Modality Encoder Representations from Transformers](https://mlsanity.com/paper/1908.07490/) (2019)
- [Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answering](https://mlsanity.com/paper/1707.07998/) (2017)
- [VisualBERT: A Simple and Performant Baseline for Vision and Language](https://mlsanity.com/paper/1908.03557/) (2019)
- [Oscar: Object-Semantics Aligned Pre-training for Vision-Language Tasks](https://mlsanity.com/paper/2004.06165/) (2020)
- [Deep Modular Co-Attention Networks for Visual Question Answering](https://mlsanity.com/paper/1906.10770/) (2019)
- [Skip-Thought Vectors](https://mlsanity.com/paper/1506.06726/) (2015)
- [Multimodal Compact Bilinear Pooling for Visual Question Answering and Visual Grounding](https://mlsanity.com/paper/1606.01847/) (2016)
- [In Defense of Grid Features for Visual Question Answering](https://mlsanity.com/paper/2001.03615/) (2020)
- [MUTAN: Multimodal Tucker Fusion for Visual Question Answering](https://mlsanity.com/paper/1705.06676/) (2017)
- [DeepFashion2: A Versatile Benchmark for Detection, Pose Estimation, Segmentation and Re-Identification of Clothing Images](https://mlsanity.com/paper/1901.07973/) (2019)
- [Dynamic Fusion with Intra- and Inter- Modality Attention Flow for Visual Question Answering](https://mlsanity.com/paper/1812.05252/) (2018)
- [Compact Bilinear Pooling](https://mlsanity.com/paper/1511.06062/) (2015)
- [Fashionpedia: Ontology, Segmentation, and an Attribute Localization Dataset](https://mlsanity.com/paper/2004.12276/) (2020)
- [The iMaterialist Fashion Attribute Dataset](https://mlsanity.com/paper/1906.05750/) (2019)
- [ModaNet: A Large-Scale Street Fashion Dataset with Polygon Annotations](https://mlsanity.com/paper/1807.01394/) (2018)

## Built on by (most cited first, 6 of 6)

- [Top-Down Compression: Revisit Efficient Vision Token Projection for Visual Instruction Tuning](https://mlsanity.com/paper/2505.11945/) (2025)
- [TriplePlay: Enhancing Federated Learning with CLIP for Non-IID Data and Resource Efficiency](https://mlsanity.com/paper/2409.05347/) (2024)
- [GPT Sonograpy: Hand Gesture Decoding from Forearm Ultrasound Images via VLM](https://mlsanity.com/paper/2407.10870/) (2024)
- [ENCLIP: Ensembling and Clustering-Based Contrastive Language-Image Pretraining for Fashion Multimodal Search with Limited Data and Low-Quality Images](https://mlsanity.com/paper/2411.16096/) (2024)
- [OmniFashion: Towards Generalist Fashion Intelligence via Multi-Task Vision-Language Learning](https://mlsanity.com/paper/2603.02658/) (2026)
- [FashionKG-RAG: Knowledge Graph-Enhanced Retrieval-Augmented Generation for Fashion Question Answering](https://mlsanity.com/paper/2608.22688/) (2026)

## Abstract (arXiv)

Humans apprehend the world through various sensory modalities, yet language is their predominant communication channel. Machine learning systems need to draw on the same multimodal richness to have informed discourses with humans in natural language; this is particularly true for systems specialized in visually-dense information, such as dialogue, recommendation, and search engines for clothing. To this end, we train a visual question answering (VQA) system to answer complex natural language questions about apparel in fashion photoshoot images. The key to the successful training of our VQA model is the automatic creation of a visual question-answering dataset with 168 million samples from item attributes of 207 thousand images using diverse templates. The sample generation employs a strategy that considers the difficulty of the question-answer pairs to emphasize challenging concepts. Contrary to the recent trends in using several datasets for pretraining the visual question answering models, we focused on keeping the dataset fixed while training various models from scratch to isolate the improvements from model architecture changes. We see that using the same transformer for encoding the question and decoding the answer, as in language models, achieves maximum accuracy, showing that visual language models (VLMs) make the best visual question answering systems for our dataset. The accuracy of the best model surpasses the human expert level, even when answering human-generated questions that are not confined to the template formats. Our approach for generating a large-scale multimodal domain-specific dataset provides a path for training specialized models capable of communicating in natural language. The training of such domain-expert models, e.g., our fashion VLM model, cannot rely solely on the large-scale general-purpose datasets collected from the web.