# PICASSO: Unleashing the Potential of GPU-centric Training for Wide-and-deep Recommender Systems

arXiv: https://arxiv.org/abs/2204.04903 · HTML: https://mlsanity.com/paper/2204.04903/

## Key facts

- Cited by 17 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 11 Apr 2022.

## Built on by (most cited first, 17 of 17)

- [Towards Understanding the Overfitting Phenomenon of Deep Click-Through Rate Prediction Models](https://mlsanity.com/paper/2209.06053/) (2022)
- [Enhancing Taobao Display Advertising with Multimodal Representations: Challenges, Approaches and Insights](https://mlsanity.com/paper/2407.19467/) (2024)
- [Joint Optimization of Ranking and Calibration with Contextualized Hybrid Model](https://mlsanity.com/paper/2208.06164/) (2022)
- [KEEP: An Industrial Pre-Training Framework for Online Recommendation via Knowledge Extraction and Plugging](https://mlsanity.com/paper/2208.10174/) (2022)
- [Rec4Ad: A Free Lunch to Mitigate Sample Selection Bias for Ads CTR Prediction in Taobao](https://mlsanity.com/paper/2306.03527/) (2023)
- [COPR: Consistency-Oriented Pre-Ranking for Online Advertising](https://mlsanity.com/paper/2306.03516/) (2023)
- [Capturing Conversion Rate Fluctuation during Sales Promotions: A Novel Historical Data Reuse Approach](https://mlsanity.com/paper/2305.12837/) (2023)
- [Deep Task-specific Bottom Representation Network for Multi-Task Recommendation](https://mlsanity.com/paper/2308.05996/) (2023)
- [GBA: A Tuning-free Approach to Switch between Synchronous and Asynchronous Training for Recommendation Model](https://mlsanity.com/paper/2205.11048/) (2022)
- [Hybrid Contrastive Constraints for Multi-Scenario Ad Ranking](https://mlsanity.com/paper/2302.02636/) (2023)
- [G-Meta: Distributed Meta Learning in GPU Clusters for Large-Scale Recommender Systems](https://mlsanity.com/paper/2401.04338/) (2024)
- [AdaptDHM: Adaptive Distribution Hierarchical Model for Multi-Domain CTR Prediction](https://mlsanity.com/paper/2211.12105/) (2022)
- [ESMC: Entire Space Multi-Task Model for Post-Click Conversion Rate via Parameter Constraint](https://mlsanity.com/paper/2307.09193/) (2023)
- [RecIS: Sparse to Dense, A Unified Training Framework for Recommendation Models](https://mlsanity.com/paper/2509.20883/) (2025)
- [Adaptive Regularization for Large-Scale Sparse Feature Embedding Models](https://mlsanity.com/paper/2511.06374/) (2025)
- [NestPipe: Large-Scale Recommendation Training on 1,500+ Accelerators via Nested Pipelining](https://mlsanity.com/paper/2604.06956/) (2026)
- [Mitigating Early Training Collapse in CTR Models](https://mlsanity.com/paper/2607.09696/) (2026)

## Abstract (arXiv)

The development of personalized recommendation has significantly improved the accuracy of information matching and the revenue of e-commerce platforms. Recently, it has 2 trends: 1) recommender systems must be trained timely to cope with ever-growing new products and ever-changing user interests from online marketing and social network; 2) SOTA recommendation models introduce DNN modules to improve prediction accuracy. Traditional CPU-based recommender systems cannot meet these two trends, and GPU- centric training has become a trending approach. However, we observe that GPU devices in training recommender systems are underutilized, and they cannot attain an expected throughput improvement as what it has achieved in CV and NLP areas. This issue can be explained by two characteristics of these recommendation models: First, they contain up to a thousand input feature fields, introducing fragmentary and memory-intensive operations; Second, the multiple constituent feature interaction submodules introduce substantial small-sized compute kernels. To remove this roadblock to the development of recommender systems, we propose a novel framework named PICASSO to accelerate the training of recommendation models on commodity hardware. Specifically, we conduct a systematic analysis to reveal the bottlenecks encountered in training recommendation models. We leverage the model structure and data distribution to unleash the potential of hardware through our packing, interleaving, and caching optimization. Experiments show that PICASSO increases the hardware utilization by an order of magnitude on the basis of SOTA baselines and brings up to 6x throughput improvement for a variety of industrial recommendation models. Using the same hardware budget in production, PICASSO on average shortens the walltime of daily training tasks by 7 hours, significantly reducing the delay of continuous delivery.