# MDDL: A Framework for Reinforcement Learning-based Position Allocation in Multi-Channel Feed

arXiv: https://arxiv.org/abs/2304.09087 · HTML: https://mlsanity.com/paper/2304.09087/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 15 of its 24 references that are in the map.
- First posted on arXiv 17 Apr 2023.

## Builds on (most cited first, 15 of 15)

- [Conservative Q-Learning for Offline Reinforcement Learning](https://mlsanity.com/paper/2006.04779/) (2020)
- [Deep Reinforcement Learning with Double Q-learning](https://mlsanity.com/paper/1509.06461/) (2015)
- [Generative Adversarial Imitation Learning](https://mlsanity.com/paper/1606.03476/) (2016)
- [An Algorithmic Perspective on Imitation Learning](https://mlsanity.com/paper/1811.06711/) (2018)
- [Randomized Ensembled Double Q-Learning: Learning Fast Without a Model](https://mlsanity.com/paper/2101.05982/) (2021)
- [Maxmin Q-learning: Controlling the Estimation Bias of Q-learning](https://mlsanity.com/paper/2002.06487/) (2020)
- [DEAR: Deep Reinforcement Learning for Online Advertising Impression in Recommender Systems](https://mlsanity.com/paper/1909.03602/) (2019)
- [MiNet: Mixed Interest Network for Cross-Domain Click-Through Rate Prediction](https://mlsanity.com/paper/2008.02974/) (2020)
- [Jointly Learning to Recommend and Advertise](https://mlsanity.com/paper/2003.00097/) (2020)
- [Cross DQN: Cross Deep Q Network for Ads Allocation in Feed](https://mlsanity.com/paper/2109.04353/) (2021)
- [The Effect of Multi-step Methods on Overestimation in Deep Reinforcement Learning](https://mlsanity.com/paper/2006.12692/) (2020)
- [Deep Page-Level Interest Network in Reinforcement Learning for Ads Allocation](https://mlsanity.com/paper/2204.00377/) (2022)
- [Deep Reinforcement Learning with Weighted Q-Learning](https://mlsanity.com/paper/2003.09280/) (2020)
- [Hybrid Transfer in Deep Reinforcement Learning for Ads Allocation](https://mlsanity.com/paper/2204.11589/) (2022)
- [Learning List-wise Representation in Reinforcement Learning for Ads Allocation with Multiple Auxiliary Tasks](https://mlsanity.com/paper/2204.00888/) (2022)

## Built on by (most cited first, 1 of 1)

- [Deep Automated Mechanism Design for Integrating Ad Auction and Allocation in Feed](https://mlsanity.com/paper/2401.01656/) (2024)

## Abstract (arXiv)

Nowadays, the mainstream approach in position allocation system is to utilize a reinforcement learning model to allocate appropriate locations for items in various channels and then mix them into the feed. There are two types of data employed to train reinforcement learning (RL) model for position allocation, named strategy data and random data. Strategy data is collected from the current online model, it suffers from an imbalanced distribution of state-action pairs, resulting in severe overestimation problems during training. On the other hand, random data offers a more uniform distribution of state-action pairs, but is challenging to obtain in industrial scenarios as it could negatively impact platform revenue and user experience due to random exploration. As the two types of data have different distributions, designing an effective strategy to leverage both types of data to enhance the efficacy of the RL model training has become a highly challenging problem. In this study, we propose a framework named Multi-Distribution Data Learning (MDDL) to address the challenge of effectively utilizing both strategy and random data for training RL models on mixed multi-distribution data. Specifically, MDDL incorporates a novel imitation learning signal to mitigate overestimation problems in strategy data and maximizes the RL signal for random data to facilitate effective learning. In our experiments, we evaluated the proposed MDDL framework in a real-world position allocation system and demonstrated its superior performance compared to the previous baseline. MDDL has been fully deployed on the Meituan food delivery platform and currently serves over 300 million users.