# Towards the Better Ranking Consistency: A Multi-task Learning Framework for Early Stage Ads Ranking

arXiv: https://arxiv.org/abs/2307.11096 · HTML: https://mlsanity.com/paper/2307.11096/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 11 of its 24 references that are in the map.
- First posted on arXiv 12 Jul 2023.

## Builds on (most cited first, 11 of 11)

- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [Multi-Task Learning Using Uncertainty to Weigh Losses for Scene Geometry and Semantics](https://mlsanity.com/paper/1705.07115/) (2017)
- [Gradient Surgery for Multi-Task Learning](https://mlsanity.com/paper/2001.06782/) (2020)
- [Multi-Task Learning as Multi-Objective Optimization](https://mlsanity.com/paper/1810.04650/) (2018)
- [Deep Interest Network for Click-Through Rate Prediction](https://mlsanity.com/paper/1706.06978/) (2017)
- [Deep Learning Recommendation Model for Personalization and Recommendation Systems](https://mlsanity.com/paper/1906.00091/) (2019)
- [Bias and Debias in Recommender System: A Survey and Future Directions](https://mlsanity.com/paper/2010.03240/) (2020)
- [Entire Space Multi-Task Model: An Effective Approach for Estimating Post-Click Conversion Rate](https://mlsanity.com/paper/1804.07931/) (2018)
- [A Survey on Negative Transfer](https://mlsanity.com/paper/2009.00909/) (2020)
- [Towards a Better Tradeoff between Effectiveness and Efficiency in Pre-Ranking: A Learnable Feature Selection based Approach](https://mlsanity.com/paper/2105.07706/) (2021)
- [On Ranking Consistency of Pre-ranking Stage](https://mlsanity.com/paper/2205.01289/) (2022)

## Built on by (most cited first, 7 of 7)

- [Async Learned User Embeddings for Ads Delivery Optimization](https://mlsanity.com/paper/2406.05898/) (2024)
- [Adaptive Neural Ranking Framework: Toward Maximized Business Goal for Cascade Ranking Systems](https://mlsanity.com/paper/2310.10462/) (2023)
- [MultiBalance: Multi-Objective Gradient Balancing in Industrial-Scale Multi-Task Recommendation System](https://mlsanity.com/paper/2411.11871/) (2024)
- [Bipartite Ranking From Multiple Labels: On Loss Versus Label Aggregation](https://mlsanity.com/paper/2504.11284/) (2025)
- [MTMD: A Multi-Task Multi-Domain Framework for Unified Ad Lightweight Ranking at Pinterest](https://mlsanity.com/paper/2510.09857/) (2025)
- [Compute Only Once: UG-Separation for Efficient Large Recommendation Models](https://mlsanity.com/paper/2602.10455/) (2026)
- [A Unified Knowledge-Distillation and Semi-Supervised Learning Framework to Improve Industrial Ads Delivery Systems](https://mlsanity.com/paper/2502.06834/) (2025)

## Abstract (arXiv)

Dividing ads ranking system into retrieval, early, and final stages is a common practice in large scale ads recommendation to balance the efficiency and accuracy. The early stage ranking often uses efficient models to generate candidates out of a set of retrieved ads. The candidates are then fed into a more computationally intensive but accurate final stage ranking system to produce the final ads recommendation. As the early and final stage ranking use different features and model architectures because of system constraints, a serious ranking consistency issue arises where the early stage has a low ads recall, i.e., top ads in the final stage are ranked low in the early stage. In order to pass better ads from the early to the final stage ranking, we propose a multi-task learning framework for early stage ranking to capture multiple final stage ranking components (i.e. ads clicks and ads quality events) and their task relations. With our multi-task learning framework, we can not only achieve serving cost saving from the model consolidation, but also improve the ads recall and ranking consistency. In the online A/B testing, our framework achieves significantly higher click-through rate (CTR), conversion rate (CVR), total value and better ads-quality (e.g. reduced ads cross-out rate) in a large scale industrial ads ranking system.