# Enhancing Taobao Display Advertising with Multimodal Representations: Challenges, Approaches and Insights

arXiv: https://arxiv.org/abs/2407.19467 · HTML: https://mlsanity.com/paper/2407.19467/

## Key facts

- Cited by 23 papers in the map (data through 14 Oct 2026).
- Builds on 30 of its 38 references that are in the map.
- First posted on arXiv 28 Jul 2024.

## Builds on (most cited first, 30 of 30)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Representation Learning with Contrastive Predictive Coding](https://mlsanity.com/paper/1807.03748/) (2018)
- [Momentum Contrast for Unsupervised Visual Representation Learning](https://mlsanity.com/paper/1911.05722/) (2019)
- [On Calibration of Modern Neural Networks](https://mlsanity.com/paper/1706.04599/) (2017)
- [FaceNet: A Unified Embedding for Face Recognition and Clustering](https://mlsanity.com/paper/1503.03832/) (2015)
- [Deep Sets](https://mlsanity.com/paper/1703.06114/) (2017)
- [Wide & Deep Learning for Recommender Systems](https://mlsanity.com/paper/1606.07792/) (2016)
- [Deep Interest Network for Click-Through Rate Prediction](https://mlsanity.com/paper/1706.06978/) (2017)
- [Ups and Downs: Modeling the Visual Evolution of Fashion Trends with One-Class Collaborative Filtering](https://mlsanity.com/paper/1602.01585/) (2016)
- [Deep Interest Evolution Network for Click-Through Rate Prediction](https://mlsanity.com/paper/1809.03672/) (2018)
- [Embedding-based Retrieval in Facebook Search](https://mlsanity.com/paper/2006.11632/) (2020)
- [Where to Go Next for Recommender Systems? ID- vs. Modality-based Recommender Models Revisited](https://mlsanity.com/paper/2303.13835/) (2023)
- [Learning Tree-based Deep Model for Recommender Systems](https://mlsanity.com/paper/1801.02294/) (2018)
- [One Model to Serve All: Star Topology Adaptive Recommender for Multi-Domain CTR Prediction](https://mlsanity.com/paper/2101.11427/) (2021)
- [PinnerSage: Multi-Modal User Embedding Framework for Recommendations at Pinterest](https://mlsanity.com/paper/2007.03634/) (2020)
- [Optimized Cost per Click in Taobao Display Advertising](https://mlsanity.com/paper/1703.02091/) (2017)
- [Better Generalization with Semantic IDs: A Case Study in Ranking for Recommendations](https://mlsanity.com/paper/2306.08121/) (2023)
- [Towards Understanding the Overfitting Phenomenon of Deep Click-Through Rate Prediction Models](https://mlsanity.com/paper/2209.06053/) (2022)
- [Deep CTR Prediction in Display Advertising](https://mlsanity.com/paper/1609.06018/) (2016)
- [COLD: Towards the Next Generation of Pre-Ranking System](https://mlsanity.com/paper/2007.16122/) (2020)
- [Joint Optimization of Ranking and Calibration with Contextualized Hybrid Model](https://mlsanity.com/paper/2208.06164/) (2022)
- [Real Negatives Matter: Continuous Training with Real Negatives for Delayed Feedback Modeling](https://mlsanity.com/paper/2104.14121/) (2021)
- [PICASSO: Unleashing the Potential of GPU-centric Training for Wide-and-deep Recommender Systems](https://mlsanity.com/paper/2204.04903/) (2022)
- [Image Matters: Visually modeling user behaviors using Advanced Model Server](https://mlsanity.com/paper/1711.06505/) (2017)
- [KEEP: An Industrial Pre-Training Framework for Online Recommendation via Knowledge Extraction and Plugging](https://mlsanity.com/paper/2208.10174/) (2022)
- [COPR: Consistency-Oriented Pre-Ranking for Online Advertising](https://mlsanity.com/paper/2306.03516/) (2023)
- [Capturing Conversion Rate Fluctuation during Sales Promotions: A Novel Historical Data Reuse Approach](https://mlsanity.com/paper/2305.12837/) (2023)
- [End-to-End Image-Based Fashion Recommendation](https://mlsanity.com/paper/2205.02923/) (2022)
- [Adversarial Gradient Driven Exploration for Deep Click-Through Rate Prediction](https://mlsanity.com/paper/2112.11136/) (2021)
- [Images Don't Lie: Transferring Deep Visual Semantic Features to Large-Scale Multimodal Learning to Rank](https://mlsanity.com/paper/1511.06746/) (2015)

## Built on by (most cited first, 23 of 23)

- [A Survey on Large Language Models in Multimodal Recommender Systems](https://mlsanity.com/paper/2505.09777/) (2025)
- [MUSE: A Simple Yet Effective Multimodal Search-Based Framework for Lifelong User Interest Modeling](https://mlsanity.com/paper/2512.07216/) (2025)
- [See Beyond a Single View: Multi-Attribution Learning Leads to Better Conversion Rate Prediction](https://mlsanity.com/paper/2508.15217/) (2025)
- [Pre-train, Align, and Disentangle: Empowering Sequential Recommendation with Large Language Models](https://mlsanity.com/paper/2412.04107/) (2024)
- [Diffusion-based Multi-modal Synergy Interest Network for Click-through Rate Prediction](https://mlsanity.com/paper/2508.21460/) (2025)
- [RecIS: Sparse to Dense, A Unified Training Framework for Recommendation Models](https://mlsanity.com/paper/2509.20883/) (2025)
- [AIF: Asynchronous Inference Framework for Cost-Effective Pre-Ranking](https://mlsanity.com/paper/2511.12934/) (2025)
- [IDProxy: CTR Prediction with Multimodal LLMs for Cold-Start Recommendation at Xiaohongshu](https://mlsanity.com/paper/2603.01590/) (2026)
- [KARMA: Knowledge-Action Regularized Multimodal Alignment for Personalized Search at Taobao](https://mlsanity.com/paper/2603.22779/) (2026)
- [A Metric for MLLM Alignment in Large-scale Recommendation](https://mlsanity.com/paper/2508.04963/) (2025)
- [Lightweight Auto-bidding based on Traffic Prediction in Live Advertising](https://mlsanity.com/paper/2508.06069/) (2025)
- [MISS: Multi-Modal Tree Indexing and Searching with Lifelong Sequential Behavior for Retrieval Recommendation](https://mlsanity.com/paper/2508.14515/) (2025)
- [RQ-GMM: Residual Quantized Gaussian Mixture Model for Multimodal Semantic Discretization in CTR Prediction](https://mlsanity.com/paper/2602.12593/) (2026)
- [MAC: A Conversion Rate Prediction Benchmark Featuring Labels Under Multiple Attribution Mechanisms](https://mlsanity.com/paper/2603.02184/) (2026)
- [VisualLens: Personalization through Task-Agnostic Visual History](https://mlsanity.com/paper/2411.16034/) (2024)
- [Bidding-Aware Retrieval for Multi-Stage Consistency in Online Advertising](https://mlsanity.com/paper/2508.05206/) (2025)
- [MATT-CTR: Unleashing a Model-Agnostic Test-Time Paradigm for CTR Prediction with Confidence-Guided Inference Paths](https://mlsanity.com/paper/2510.08932/) (2025)
- [Conversion rate prediction in online advertising: modeling techniques, performance evaluation and future directions](https://mlsanity.com/paper/2512.01171/) (2025)
- [Revisiting Content-Based Music Recommendation: Efficient Feature Aggregation from Large-Scale Music Models](https://mlsanity.com/paper/2604.20847/) (2026)
- [FLUID: From Ephemeral IDs to Multimodal Semantic Codes for Industrial-Scale Livestreaming Recommendation](https://mlsanity.com/paper/2605.21832/) (2026)
- [MMRM: A Multiplex Multimodal Representation Model for Product Ranking in E-commerce Search](https://mlsanity.com/paper/2607.11030/) (2026)
- [PaletteID: Prototype-Composed Semantic Identifiers for Multimodal CTR Prediction](https://mlsanity.com/paper/2607.29000/) (2026)
- [GALA: Generative Aligned Learning for Adaptive Multimodal Representation in the Taobao Shangou Recommender System](https://mlsanity.com/paper/2607.29213/) (2026)

## Abstract (arXiv)

Despite the recognized potential of multimodal data to improve model accuracy, many large-scale industrial recommendation systems, including Taobao display advertising system, predominantly depend on sparse ID features in their models. In this work, we explore approaches to leverage multimodal data to enhance the recommendation accuracy. We start from identifying the key challenges in adopting multimodal data in a manner that is both effective and cost-efficient for industrial systems. To address these challenges, we introduce a two-phase framework, including: 1) the pre-training of multimodal representations to capture semantic similarity, and 2) the integration of these representations with existing ID-based models. Furthermore, we detail the architecture of our production system, which is designed to facilitate the deployment of multimodal representations. Since the integration of multimodal representations in mid-2023, we have observed significant performance improvements in Taobao display advertising system. We believe that the insights we have gathered will serve as a valuable resource for practitioners seeking to leverage multimodal data in their systems.