# Learning to Collaborate: Multi-Scenario Ranking via Multi-Agent Reinforcement Learning

arXiv: https://arxiv.org/abs/1809.06260 · HTML: https://mlsanity.com/paper/1809.06260/

## Key facts

- Cited by 23 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 17 Sep 2018.

## Built on by (most cited first, 23 of 23)

- [Shall We Team Up: Exploring Spontaneous Cooperation of Competing LLM Agents](https://mlsanity.com/paper/2402.12327/) (2024)
- [Multi-Task Recommendations with Reinforcement Learning](https://mlsanity.com/paper/2302.03328/) (2023)
- [A Survey of Deep Reinforcement Learning in Recommender Systems: A Systematic Review and Future Directions](https://mlsanity.com/paper/2109.03540/) (2021)
- [Cross DQN: Cross Deep Q Network for Ads Allocation in Feed](https://mlsanity.com/paper/2109.04353/) (2021)
- [A Survey on Deep Reinforcement Learning for Data Processing and Analytics](https://mlsanity.com/paper/2108.04526/) (2021)
- [Reinforcement Learning to Optimize Lifetime Value in Cold-Start Recommendation](https://mlsanity.com/paper/2108.09141/) (2021)
- [Deep Page-Level Interest Network in Reinforcement Learning for Ads Allocation](https://mlsanity.com/paper/2204.00377/) (2022)
- [FRAMM: Fair Ranking with Missing Modalities for Clinical Trial Site Selection](https://mlsanity.com/paper/2305.19407/) (2023)
- [Hybrid Transfer in Deep Reinforcement Learning for Ads Allocation](https://mlsanity.com/paper/2204.11589/) (2022)
- [Multi-Graph based Multi-Scenario Recommendation in Large-scale Online Video Services](https://mlsanity.com/paper/2205.02446/) (2022)
- [Hybrid Contrastive Constraints for Multi-Scenario Ad Ranking](https://mlsanity.com/paper/2302.02636/) (2023)
- [Learning List-wise Representation in Reinforcement Learning for Ads Allocation with Multiple Auxiliary Tasks](https://mlsanity.com/paper/2204.00888/) (2022)
- [Collaborative Reasoning on Multi-Modal Semantic Graphs for Video-Grounded Dialogue Generation](https://mlsanity.com/paper/2210.12460/) (2022)
- [DARLR: Dual-Agent Offline Reinforcement Learning for Recommender Systems with Dynamic Reward](https://mlsanity.com/paper/2505.07257/) (2025)
- [Deep Reinforcement Learning for Ranking Utility Tuning in the Ad Recommender System at Pinterest](https://mlsanity.com/paper/2509.05292/) (2025)
- [SINGA-Easy: An Easy-to-Use Framework for MultiModal Analysis](https://mlsanity.com/paper/2108.02572/) (2021)
- [VisualTextRank: Unsupervised Graph-based Content Extraction for Automating Ad Text to Image Search](https://mlsanity.com/paper/2108.02725/) (2021)
- [Text is NOT Enough: Integrating Visual Impressions into Open-domain Dialogue Generation](https://mlsanity.com/paper/2109.05778/) (2021)
- [Discovering Personalized Semantics for Soft Attributes in Recommender Systems using Concept Activation Vectors](https://mlsanity.com/paper/2202.02830/) (2022)
- [Reinforced MOOCs Concept Recommendation in Heterogeneous Information Networks](https://mlsanity.com/paper/2203.11011/) (2022)
- [Explanation-Guided Fairness Testing through Genetic Algorithm](https://mlsanity.com/paper/2205.08335/) (2022)
- [Using Graph Neural Networks for Program Termination](https://mlsanity.com/paper/2207.14648/) (2022)
- [Tribe or Not? Critical Inspection of Group Differences Using TribalGram](https://mlsanity.com/paper/2303.09664/) (2023)

## Abstract (arXiv)

Ranking is a fundamental and widely studied problem in scenarios such as search, advertising, and recommendation. However, joint optimization for multi-scenario ranking, which aims to improve the overall performance of several ranking strategies in different scenarios, is rather untouched. Separately optimizing each individual strategy has two limitations. The first one is lack of collaboration between scenarios meaning that each strategy maximizes its own objective but ignores the goals of other strategies, leading to a sub-optimal overall performance. The second limitation is the inability of modeling the correlation between scenarios meaning that independent optimization in one scenario only uses its own user data but ignores the context in other scenarios. In this paper, we formulate multi-scenario ranking as a fully cooperative, partially observable, multi-agent sequential decision problem. We propose a novel model named Multi-Agent Recurrent Deterministic Policy Gradient (MA-RDPG) which has a communication component for passing messages, several private actors (agents) for making actions for ranking, and a centralized critic for evaluating the overall performance of the co-working actors. Each scenario is treated as an agent (actor). Agents collaborate with each other by sharing a global action-value function (the critic) and passing messages that encodes historical information across scenarios. The model is evaluated with online settings on a large E-commerce platform. Results show that the proposed model exhibits significant improvements against baselines in terms of the overall performance.