# M2FN: Multi-step Modality Fusion for Advertisement Image Assessment

arXiv: https://arxiv.org/abs/2102.00441 · HTML: https://mlsanity.com/paper/2102.00441/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 20 of its 38 references that are in the map.
- First posted on arXiv 31 Jan 2021.

## Builds on (most cited first, 20 of 20)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [The Unreasonable Effectiveness of Deep Features as a Perceptual Metric](https://mlsanity.com/paper/1801.03924/) (2018)
- [Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localization](https://mlsanity.com/paper/1610.02391/) (2016)
- [VQA: Visual Question Answering](https://mlsanity.com/paper/1505.00468/) (2015)
- [Show, Attend and Tell: Neural Image Caption Generation with Visual Attention](https://mlsanity.com/paper/1502.03044/) (2015)
- [NIMA: Neural Image Assessment](https://mlsanity.com/paper/1709.05424/) (2017)
- [Character Region Awareness for Text Detection](https://mlsanity.com/paper/1904.01941/) (2019)
- [Multimodal Compact Bilinear Pooling for Visual Question Answering and Visual Grounding](https://mlsanity.com/paper/1606.01847/) (2016)
- [What Is Wrong With Scene Text Recognition Model Comparisons? Dataset and Model Analysis](https://mlsanity.com/paper/1904.01906/) (2019)
- [Modulating early visual processing by language](https://mlsanity.com/paper/1707.00683/) (2017)
- [MFAS: Multimodal Fusion Architecture Search](https://mlsanity.com/paper/1903.06496/) (2019)
- [NSML: Meet the MLaaS platform with a real-world case study](https://mlsanity.com/paper/1810.09957/) (2018)
- [Improved Fusion of Visual and Language Representations by Dense Symmetric Co-Attention for Visual Question Answering](https://mlsanity.com/paper/1804.00775/) (2018)
- [Image Aesthetic Assessment: An Experimental Survey](https://mlsanity.com/paper/1610.00838/) (2016)
- [NSML: A Machine Learning Platform That Enables You to Focus on Your Models](https://mlsanity.com/paper/1712.05902/) (2017)
- [Deep CTR Prediction in Display Advertising](https://mlsanity.com/paper/1609.06018/) (2016)
- [Multimodal Dual Attention Memory for Video Story Question Answering](https://mlsanity.com/paper/1809.07999/) (2018)
- [Aesthetic-based Clothing Recommendation](https://mlsanity.com/paper/1809.05822/) (2018)
- [CHOPT : Automated Hyperparameter Optimization Framework for Cloud-Based Machine Learning Platforms](https://mlsanity.com/paper/1810.03527/) (2018)

## Abstract (arXiv)

Assessing advertisements, specifically on the basis of user preferences and ad quality, is crucial to the marketing industry. Although recent studies have attempted to use deep neural networks for this purpose, these studies have not utilized image-related auxiliary attributes, which include embedded text frequently found in ad images. We, therefore, investigated the influence of these attributes on ad image preferences. First, we analyzed large-scale real-world ad log data and, based on our findings, proposed a novel multi-step modality fusion network (M2FN) that determines advertising images likely to appeal to user preferences. Our method utilizes auxiliary attributes through multiple steps in the network, which include conditional batch normalization-based low-level fusion and attention-based high-level fusion. We verified M2FN on the AVA dataset, which is widely used for aesthetic image assessment, and then demonstrated that M2FN can achieve state-of-the-art performance in preference prediction using a real-world ad dataset with rich auxiliary attributes.