# Learning-To-Ensemble by Contextual Rank Aggregation in E-Commerce

arXiv: https://arxiv.org/abs/2107.08598 · HTML: https://mlsanity.com/paper/2107.08598/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 14 of its 39 references that are in the map.
- First posted on arXiv 19 Jul 2021.

## Builds on (most cited first, 14 of 14)

- [Exploration by Random Network Distillation](https://mlsanity.com/paper/1810.12894/) (2018)
- [DeepFM: A Factorization-Machine based Neural Network for CTR Prediction](https://mlsanity.com/paper/1703.04247/) (2017)
- [Deep & Cross Network for Ad Click Predictions](https://mlsanity.com/paper/1708.05123/) (2017)
- [Scalable Bayesian Optimization Using Deep Neural Networks](https://mlsanity.com/paper/1502.05700/) (2015)
- [Entire Space Multi-Task Model: An Effective Approach for Estimating Post-Click Conversion Rate](https://mlsanity.com/paper/1804.07931/) (2018)
- [Unbiased Learning-to-Rank with Biased Feedback](https://mlsanity.com/paper/1608.04468/) (2016)
- [Learning Search Space Partition for Black-box Optimization using Monte Carlo Tree Search](https://mlsanity.com/paper/2007.00708/) (2020)
- [Structure Discovery in Nonparametric Regression through Compositional Kernel Search](https://mlsanity.com/paper/1302.4922/) (2013)
- [RecoGym: A Reinforcement Learning Environment for the problem of Product Recommendation in Online Advertising](https://mlsanity.com/paper/1808.00720/) (2018)
- [Ensemble Knowledge Distillation for CTR Prediction](https://mlsanity.com/paper/2011.04106/) (2020)
- [Black-Box Optimization Revisited: Improving Algorithm Selection Wizards through Massive Benchmarking](https://mlsanity.com/paper/2010.04542/) (2020)
- [Rank Aggregation: New Bounds for MCx](https://mlsanity.com/paper/1510.00738/) (2015)
- [Efficient Rank Aggregation via Lehmer Codes](https://mlsanity.com/paper/1701.09083/) (2017)
- [Multivariate Spearman's rho for aggregating ranks using copulas](https://mlsanity.com/paper/1410.4391/) (2014)

## Built on by (most cited first, 1 of 1)

- [Self-EvolveRec: Self-Evolving Recommender Systems with LLM-based Directional Feedback](https://mlsanity.com/paper/2602.12612/) (2026)

## Abstract (arXiv)

Ensemble models in E-commerce combine predictions from multiple sub-models for ranking and revenue improvement. Industrial ensemble models are typically deep neural networks, following the supervised learning paradigm to infer conversion rate given inputs from sub-models. However, this process has the following two problems. Firstly, the point-wise scoring approach disregards the relationships between items and leads to homogeneous displayed results, while diversified display benefits user experience and revenue. Secondly, the learning paradigm focuses on the ranking metrics and does not directly optimize the revenue. In our work, we propose a new Learning-To-Ensemble (LTE) framework RAEGO, which replaces the ensemble model with a contextual Rank Aggregator (RA) and explores the best weights of sub-models by the Evaluator-Generator Optimization (EGO). To achieve the best online performance, we propose a new rank aggregation algorithm TournamentGreedy as a refinement of classic rank aggregators, which also produces the best average weighted Kendall Tau Distance (KTD) amongst all the considered algorithms with quadratic time complexity. Under the assumption that the best output list should be Pareto Optimal on the KTD metric for sub-models, we show that our RA algorithm has higher efficiency and coverage in exploring the optimal weights. Combined with the idea of Bayesian Optimization and gradient descent, we solve the online contextual Black-Box Optimization task that finds the optimal weights for sub-models given a chosen RA model. RA-EGO has been deployed in our online system and has improved the revenue significantly.