# FashionBERT: Text and Image Matching with Adaptive Loss for Cross-modal Retrieval

arXiv: https://arxiv.org/abs/2005.09801 · HTML: https://mlsanity.com/paper/2005.09801/

## Key facts

- Cited by 55 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 20 May 2020.

## Built on by (most cited first, 30 of 55)

- [VLP: A Survey on Vision-Language Pre-training](https://mlsanity.com/paper/2202.09061/) (2022)
- [Large-scale Multi-Modal Pre-trained Models: A Comprehensive Survey](https://mlsanity.com/paper/2302.10035/) (2023)
- [M6: A Chinese Multimodal Pretrainer](https://mlsanity.com/paper/2103.00823/) (2021)
- [Kaleido-BERT: Vision-Language Pre-training on Fashion Domain](https://mlsanity.com/paper/2103.16110/) (2021)
- [Image-text Retrieval: A Survey on Recent Research and Development](https://mlsanity.com/paper/2203.14713/) (2022)
- [Vision-and-Language Pretrained Models: A Survey](https://mlsanity.com/paper/2204.07356/) (2022)
- [FAME-ViL: Multi-Tasking Vision-Language Model for Heterogeneous Fashion Tasks](https://mlsanity.com/paper/2303.02483/) (2023)
- [FashionViL: Fashion-Focused Vision-and-Language Representation Learning](https://mlsanity.com/paper/2207.08150/) (2022)
- [S-CLIP: Semi-supervised Vision-Language Learning using Few Specialist Captions](https://mlsanity.com/paper/2305.14095/) (2023)
- [M5Product: Self-harmonized Contrastive Learning for E-commercial Multi-modal Pretraining](https://mlsanity.com/paper/2109.04275/) (2021)
- [Progressive Learning for Image Retrieval with Hybrid-Modality Queries](https://mlsanity.com/paper/2204.11212/) (2022)
- [CommerceMM: Large-Scale Commerce MultiModal Representation Learning with Omni Retrieval](https://mlsanity.com/paper/2202.07247/) (2022)
- [Where Does the Performance Improvement Come From? -- A Reproducibility Concern about Image-Text Retrieval](https://mlsanity.com/paper/2203.03853/) (2022)
- [Knowledge Perceived Multi-modal Pretraining in E-commerce](https://mlsanity.com/paper/2109.00895/) (2021)
- [CMA-CLIP: Cross-Modality Attention CLIP for Image-Text Classification](https://mlsanity.com/paper/2112.03562/) (2021)
- [Single-Stream Multi-Level Alignment for Vision-Language Pretraining](https://mlsanity.com/paper/2203.14395/) (2022)
- [Structure Pretraining and Prompt Tuning for Knowledge Graph Transfer](https://mlsanity.com/paper/2303.03922/) (2023)
- [Masked Vision-Language Transformer in Fashion](https://mlsanity.com/paper/2210.15110/) (2022)
- [FashionSAP: Symbols and Attributes Prompt for Fine-grained Fashion Vision-Language Pre-training](https://mlsanity.com/paper/2304.05051/) (2023)
- [Leveraging Large Vision-Language Model as User Intent-aware Encoder for Composed Image Retrieval](https://mlsanity.com/paper/2412.11087/) (2024)
- [Vision Language Models in Medicine](https://mlsanity.com/paper/2503.01863/) (2025)
- [MAKE: Vision-Language Pre-training based Product Retrieval in Taobao Search](https://mlsanity.com/paper/2301.12646/) (2023)
- [SyncMask: Synchronized Attentional Masking for Fashion-centric Vision-Language Pretraining](https://mlsanity.com/paper/2404.01156/) (2024)
- [Learning Instance-Level Representation for Large-Scale Multi-Modal Pretraining in E-commerce](https://mlsanity.com/paper/2304.02853/) (2023)
- [Structure Guided Multi-modal Pre-trained Transformer for Knowledge Graph Reasoning](https://mlsanity.com/paper/2307.03591/) (2023)
- [Asking Multimodal Clarifying Questions in Mixed-Initiative Conversational Search](https://mlsanity.com/paper/2402.07742/) (2024)
- [FaD-VLP: Fashion Vision-and-Language Pre-training towards Unified Retrieval and Captioning](https://mlsanity.com/paper/2210.15028/) (2022)
- [MOON: Generative MLLM-based Multimodal Representation Learning for E-commerce Product Understanding](https://mlsanity.com/paper/2508.11999/) (2025)
- [Training and challenging models for text-guided fashion image retrieval](https://mlsanity.com/paper/2204.11004/) (2022)
- [CapEnrich: Enriching Caption Semantics for Web Images via Cross-modal Pre-trained Knowledge](https://mlsanity.com/paper/2211.09371/) (2022)

## Abstract (arXiv)

In this paper, we address the text and image matching in cross-modal retrieval of the fashion industry. Different from the matching in the general domain, the fashion matching is required to pay much more attention to the fine-grained information in the fashion images and texts. Pioneer approaches detect the region of interests (i.e., RoIs) from images and use the RoI embeddings as image representations. In general, RoIs tend to represent the "object-level" information in the fashion images, while fashion texts are prone to describe more detailed information, e.g. styles, attributes. RoIs are thus not fine-grained enough for fashion text and image matching. To this end, we propose FashionBERT, which leverages patches as image features. With the pre-trained BERT model as the backbone network, FashionBERT learns high level representations of texts and images. Meanwhile, we propose an adaptive loss to trade off multitask learning in the FashionBERT modeling. Two tasks (i.e., text and image matching and cross-modal retrieval) are incorporated to evaluate FashionBERT. On the public dataset, experiments demonstrate FashionBERT achieves significant improvements in performances than the baseline and state-of-the-art approaches. In practice, FashionBERT is applied in a concrete cross-modal retrieval application. We provide the detailed matching performance and inference efficiency analysis.