# Multi-Modal Attribute Extraction for E-Commerce

arXiv: https://arxiv.org/abs/2203.03441 · HTML: https://mlsanity.com/paper/2203.03441/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 21 of its 34 references that are in the map.
- First posted on arXiv 7 Mar 2022.

## Builds on (most cited first, 21 of 21)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Densely Connected Convolutional Networks](https://mlsanity.com/paper/1608.06993/) (2016)
- [Layer Normalization](https://mlsanity.com/paper/1607.06450/) (2016)
- [Convolutional Neural Networks for Sentence Classification](https://mlsanity.com/paper/1408.5882/) (2014)
- [Multimodal Machine Learning: A Survey and Taxonomy](https://mlsanity.com/paper/1705.09406/) (2017)
- [Bag of Tricks for Efficient Text Classification](https://mlsanity.com/paper/1607.01759/) (2016)
- [Gated Multimodal Units for Information Fusion](https://mlsanity.com/paper/1702.01992/) (2017)
- [Supervised Multimodal Bitransformers for Classifying Images and Text](https://mlsanity.com/paper/1909.02950/) (2019)
- [CentralNet: a Multilayer Approach for Multimodal Fusion](https://mlsanity.com/paper/1808.07275/) (2018)
- [MFAS: Multimodal Fusion Architecture Search](https://mlsanity.com/paper/1903.06496/) (2019)
- [Efficient Large-Scale Multi-Modal Classification](https://mlsanity.com/paper/1802.02892/) (2018)
- [Multimodal Joint Attribute Prediction and Value Extraction for E-commerce Product](https://mlsanity.com/paper/2009.07162/) (2020)
- [A Network Perspective on Stratification of Multi-Label Data](https://mlsanity.com/paper/1704.08756/) (2017)
- [Are we pretraining it right? Digging deeper into visio-linguistic pretraining](https://mlsanity.com/paper/2004.08744/) (2020)
- [BM-NAS: Bilevel Multimodal Neural Architecture Search](https://mlsanity.com/paper/2104.09379/) (2021)
- [Is a picture worth a thousand words? A Deep Multi-Modal Fusion Architecture for Product Classification in e-commerce](https://mlsanity.com/paper/1611.09534/) (2016)
- [Product Characterisation towards Personalisation: Learning Attributes from Unstructured Data to Recommend Fashion Products](https://mlsanity.com/paper/1803.07679/) (2018)
- [Deep Multi-Modal Sets](https://mlsanity.com/paper/2003.01607/) (2020)
- [Training Multimodal Systems for Classification with Multiple Objectives](https://mlsanity.com/paper/2008.11450/) (2020)

## Built on by (most cited first, 5 of 5)

- [PAE: LLM-based Product Attribute Extraction for E-Commerce Fashion Trends](https://mlsanity.com/paper/2405.17533/) (2024)
- [Can GPT-4o mini and Gemini 2.0 Flash Predict Fine-Grained Fashion Product Attributes? A Zero-Shot Analysis](https://mlsanity.com/paper/2507.09950/) (2025)
- [PV2TEA: Patching Visual Modality to Textual-Established Information Extraction](https://mlsanity.com/paper/2306.01016/) (2023)
- [Question-to-Knowledge (Q2K): Multi-Agent Generation of Inspectable Facts for Product Mapping](https://mlsanity.com/paper/2509.01182/) (2025)
- [Beyond Images: Adaptive Fusion of Visual and Textual Data for Food Classification](https://mlsanity.com/paper/2308.02562/) (2023)

## Abstract (arXiv)

To improve users' experience as they navigate the myriad of options offered by online marketplaces, it is essential to have well-organized product catalogs. One key ingredient to that is the availability of product attributes such as color or material. However, on some marketplaces such as Rakuten-Ichiba, which we focus on, attribute information is often incomplete or even missing. One promising solution to this problem is to rely on deep models pre-trained on large corpora to predict attributes from unstructured data, such as product descriptive texts and images (referred to as modalities in this paper). However, we find that achieving satisfactory performance with this approach is not straightforward but rather the result of several refinements, which we discuss in this paper. We provide a detailed description of our approach to attribute extraction, from investigating strong single-modality methods, to building a solid multimodal model combining textual and visual information. One key component of our multimodal architecture is a novel approach to seamlessly combine modalities, which is inspired by our single-modality investigations. In practice, we notice that this new modality-merging method may suffer from a modality collapse issue, i.e., it neglects one modality. Hence, we further propose a mitigation to this problem based on a principled regularization scheme. Experiments on Rakuten-Ichiba data provide empirical evidence for the benefits of our approach, which has been also successfully deployed to Rakuten-Ichiba. We also report results on publicly available datasets showing that our model is competitive compared to several recent multimodal and unimodal baselines.