# Cross DQN: Cross Deep Q Network for Ads Allocation in Feed

arXiv: https://arxiv.org/abs/2109.04353 · HTML: https://mlsanity.com/paper/2109.04353/

## Key facts

- Cited by 19 papers in the map (data through 14 Oct 2026).
- Builds on 12 of its 27 references that are in the map.
- First posted on arXiv 9 Sep 2021.

## Builds on (most cited first, 12 of 12)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Dueling Network Architectures for Deep Reinforcement Learning](https://mlsanity.com/paper/1511.06581/) (2015)
- [Deep Interest Network for Click-Through Rate Prediction](https://mlsanity.com/paper/1706.06978/) (2017)
- [DEAR: Deep Reinforcement Learning for Online Advertising Impression in Recommender Systems](https://mlsanity.com/paper/1909.03602/) (2019)
- [MiNet: Mixed Interest Network for Cross-Domain Click-Through Rate Prediction](https://mlsanity.com/paper/2008.02974/) (2020)
- [Jointly Learning to Recommend and Advertise](https://mlsanity.com/paper/2003.00097/) (2020)
- [Learning to Collaborate: Multi-Scenario Ranking via Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/1809.06260/) (2018)
- [Revisit Recommender System in the Permutation Prospective](https://mlsanity.com/paper/2102.12057/) (2021)
- [GRN: Generative Rerank Network for Context-wise Recommendation](https://mlsanity.com/paper/2104.00860/) (2021)
- [Generator and Critic: A Deep Reinforcement Learning Approach for Slate Re-ranking in E-commerce](https://mlsanity.com/paper/2005.12206/) (2020)
- [Blending Advertising with Organic Content in E-Commerce: A Virtual Bids Optimization Approach](https://mlsanity.com/paper/2105.13556/) (2021)

## Built on by (most cited first, 19 of 19)

- [Deep Page-Level Interest Network in Reinforcement Learning for Ads Allocation](https://mlsanity.com/paper/2204.00377/) (2022)
- [PIER: Permutation-Level Interest-Based End-to-End Re-ranking Framework in E-commerce](https://mlsanity.com/paper/2302.03487/) (2023)
- [Hierarchically Constrained Adaptive Ad Exposure in Feeds](https://mlsanity.com/paper/2205.15759/) (2022)
- [Deep Automated Mechanism Design for Integrating Ad Auction and Allocation in Feed](https://mlsanity.com/paper/2401.01656/) (2024)
- [Hybrid Transfer in Deep Reinforcement Learning for Ads Allocation](https://mlsanity.com/paper/2204.11589/) (2022)
- [NMA: Neural Multi-slot Auctions with Externalities for Online Advertising](https://mlsanity.com/paper/2205.10018/) (2022)
- [TD3 with Reverse KL Regularizer for Offline Reinforcement Learning from Mixed Datasets](https://mlsanity.com/paper/2212.02125/) (2022)
- [HiBid: A Cross-Channel Constrained Bidding System with Budget Allocation by Hierarchical Offline Deep Reinforcement Learning](https://mlsanity.com/paper/2312.17503/) (2023)
- [Learning List-wise Representation in Reinforcement Learning for Ads Allocation with Multiple Auxiliary Tasks](https://mlsanity.com/paper/2204.00888/) (2022)
- [RL-MPCA: A Reinforcement Learning Based Multi-Phase Computation Allocation Approach for Recommender Systems](https://mlsanity.com/paper/2401.01369/) (2023)
- [Ads Supply Personalization via Doubly Robust Learning](https://mlsanity.com/paper/2410.12799/) (2024)
- [Session-Level Dynamic Ad Load Optimization using Offline Robust Reinforcement Learning](https://mlsanity.com/paper/2501.05591/) (2025)
- [Multi-task Offline Reinforcement Learning for Online Advertising in Recommender Systems](https://mlsanity.com/paper/2506.23090/) (2025)
- [MDDL: A Framework for Reinforcement Learning-based Position Allocation in Multi-Channel Feed](https://mlsanity.com/paper/2304.09087/) (2023)
- [Assortment Planning with Sponsored Products](https://mlsanity.com/paper/2402.06158/) (2024)
- [Supervised Learning-enhanced Multi-Group Actor Critic for Live Stream Allocation in Feed](https://mlsanity.com/paper/2412.10381/) (2024)
- [One Model, Two Markets: Bid-Aware Generative Recommendation](https://mlsanity.com/paper/2603.22231/) (2026)
- [Adaptive Ad Load Design for Sponsored Search Markets: Evidence, Theory, and Deployment](https://mlsanity.com/paper/2607.14418/) (2026)
- [Less Traffic, Better Outcomes: Competition-Aware Request Dispatch in Real-Time Ad Exchanges](https://mlsanity.com/paper/2608.03705/) (2026)

## Abstract (arXiv)

E-commerce platforms usually display a mixed list of ads and organic items in feed. One key problem is to allocate the limited slots in the feed to maximize the overall revenue as well as improve user experience, which requires a good model for user preference. Instead of modeling the influence of individual items on user behaviors, the arrangement signal models the influence of the arrangement of items and may lead to a better allocation strategy. However, most of previous strategies fail to model such a signal and therefore result in suboptimal performance. In addition, the percentage of ads exposed (PAE) is an important indicator in ads allocation. Excessive PAE hurts user experience while too low PAE reduces platform revenue. Therefore, how to constrain the PAE within a certain range while keeping personalized recommendation under the PAE constraint is a challenge. In this paper, we propose Cross Deep Q Network (Cross DQN) to extract the crucial arrangement signal by crossing the embeddings of different items and modeling the crossed sequence by multi-channel attention. Besides, we propose an auxiliary loss for batch-level constraint on PAE to tackle the above-mentioned challenge. Our model results in higher revenue and better user experience than state-of-the-art baselines in offline experiments. Moreover, our model demonstrates a significant improvement in the online A/B test and has been fully deployed on Meituan feed to serve more than 300 millions of customers.