# DEAR: Deep Reinforcement Learning for Online Advertising Impression in Recommender Systems

arXiv: https://arxiv.org/abs/1909.03602 · HTML: https://mlsanity.com/paper/1909.03602/

## Key facts

- Cited by 44 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 9 Sep 2019.

## Built on by (most cited first, 30 of 44)

- [A Review of Safe Reinforcement Learning: Methods, Theory and Applications](https://mlsanity.com/paper/2205.10330/) (2022)
- [Process vs. Outcome Reward: Which is Better for Agentic RAG Reinforcement Learning](https://mlsanity.com/paper/2505.14069/) (2025)
- [AutoLoss: Automated Loss Function Search in Recommendations](https://mlsanity.com/paper/2106.06713/) (2021)
- [Multi-Task Recommendations with Reinforcement Learning](https://mlsanity.com/paper/2302.03328/) (2023)
- [Generative Auto-Bidding with Value-Guided Explorations](https://mlsanity.com/paper/2504.14587/) (2025)
- [A Survey of Deep Reinforcement Learning in Recommender Systems: A Systematic Review and Future Directions](https://mlsanity.com/paper/2109.03540/) (2021)
- [Cross DQN: Cross Deep Q Network for Ads Allocation in Feed](https://mlsanity.com/paper/2109.04353/) (2021)
- [Accelerating Offline Reinforcement Learning Application in Real-Time Bidding and Recommendation: Potential Use of Simulation](https://mlsanity.com/paper/2109.08331/) (2021)
- [A Human-Centered Safe Robot Reinforcement Learning Framework with Interactive Behaviors](https://mlsanity.com/paper/2302.13137/) (2023)
- [Deep Page-Level Interest Network in Reinforcement Learning for Ads Allocation](https://mlsanity.com/paper/2204.00377/) (2022)
- [Hierarchically Constrained Adaptive Ad Exposure in Feeds](https://mlsanity.com/paper/2205.15759/) (2022)
- [Deep Automated Mechanism Design for Integrating Ad Auction and Allocation in Feed](https://mlsanity.com/paper/2401.01656/) (2024)
- [Hybrid Transfer in Deep Reinforcement Learning for Ads Allocation](https://mlsanity.com/paper/2204.11589/) (2022)
- [Benchmarking Perturbation-based Saliency Maps for Explaining Atari Agents](https://mlsanity.com/paper/2101.07312/) (2021)
- [DNS-Rec: Data-aware Neural Architecture Search for Recommender Systems](https://mlsanity.com/paper/2402.00390/) (2024)
- [Learning List-wise Representation in Reinforcement Learning for Ads Allocation with Multiple Auxiliary Tasks](https://mlsanity.com/paper/2204.00888/) (2022)
- [RL-MPCA: A Reinforcement Learning Based Multi-Phase Computation Allocation Approach for Recommender Systems](https://mlsanity.com/paper/2401.01369/) (2023)
- [Provably Efficient Algorithm for Nonstationary Low-Rank MDPs](https://mlsanity.com/paper/2308.05471/) (2023)
- [Ads Supply Personalization via Doubly Robust Learning](https://mlsanity.com/paper/2410.12799/) (2024)
- [Multi-task Offline Reinforcement Learning for Online Advertising in Recommender Systems](https://mlsanity.com/paper/2506.23090/) (2025)
- [Sequential Search with Off-Policy Reinforcement Learning](https://mlsanity.com/paper/2202.00245/) (2022)
- [XDQN: Inherently Interpretable DQN through Mimicking](https://mlsanity.com/paper/2301.03043/) (2023)
- [MDDL: A Framework for Reinforcement Learning-based Position Allocation in Multi-Channel Feed](https://mlsanity.com/paper/2304.09087/) (2023)
- [AutoAssign+: Automatic Shared Embedding Assignment in Streaming Recommendation](https://mlsanity.com/paper/2308.06965/) (2023)
- [Model-free Reinforcement Learning with Stochastic Reward Stabilization for Recommender Systems](https://mlsanity.com/paper/2308.13246/) (2023)
- [Model-enhanced Contrastive Reinforcement Learning for Sequential Recommendation](https://mlsanity.com/paper/2310.16566/) (2023)
- [WellFactor: Patient Profiling using Integrative Embedding of Healthcare Data](https://mlsanity.com/paper/2312.14129/) (2023)
- [Analyzing and Bridging the Gap between Maximizing Total Reward and Discounted Reward in Deep Reinforcement Learning](https://mlsanity.com/paper/2407.13279/) (2024)
- [FedSlate:A Federated Deep Reinforcement Learning Recommender System](https://mlsanity.com/paper/2409.14872/) (2024)
- [Guided Diffusion-based Counterfactual Augmentation for Robust Session-based Recommendation](https://mlsanity.com/paper/2410.21892/) (2024)

## Abstract (arXiv)

With the recent prevalence of Reinforcement Learning (RL), there have been tremendous interests in utilizing RL for online advertising in recommendation platforms (e.g., e-commerce and news feed sites). However, most RL-based advertising algorithms focus on optimizing ads' revenue while ignoring the possible negative influence of ads on user experience of recommended items (products, articles and videos). Developing an optimal advertising algorithm in recommendations faces immense challenges because interpolating ads improperly or too frequently may decrease user experience, while interpolating fewer ads will reduce the advertising revenue. Thus, in this paper, we propose a novel advertising strategy for the rec/ads trade-off. To be specific, we develop an RL-based framework that can continuously update its advertising strategies and maximize reward in the long run. Given a recommendation list, we design a novel Deep Q-network architecture that can determine three internally related tasks jointly, i.e., (i) whether to interpolate an ad or not in the recommendation list, and if yes, (ii) the optimal ad and (iii) the optimal location to interpolate. The experimental results based on real-world data demonstrate the effectiveness of the proposed framework.