mlsanityData to 7 Oct 2026

Session-Level Dynamic Ad Load Optimization using Offline Robust Reinforcement Learning

arXiv 2501.05591 ↗First posted 9 Jan 2025cs.LGcited by 2

Not on the map. Why?

Abstract (arXiv)

Session-level dynamic ad load optimization aims to personalize the density and types of delivered advertisements in real time during a user's online session by dynamically balancing user experience quality and ad monetization. Traditional causal learning-based approaches struggle with key technical challenges, especially in handling confounding bias and distribution shifts. In this paper, we develop an offline deep Q-network (DQN)-based framework that effectively mitigates confounding bias in dynamic systems and demonstrates more than 80% offline gains compared to the best causal learning-based production baseline. Moreover, to improve the framework's robustness against unanticipated distribution shifts, we further enhance our framework with a novel offline robust dueling DQN approach. This approach achieves more stable rewards on multiple OpenAI-Gym datasets as perturbations increase, and provides an additional 5% offline gains on real-world ad delivery data. Deployed across multiple production systems, our approach has achieved outsized topline gains. Post-launch online A/B tests have shown double-digit improvements in the engagement-ad score trade-off efficiency, significantly enhancing our platform's capability to serve both consumers and advertisers.

Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Builds on 31 of its 52 references that are in the map. First posted on arXiv 9 Jan 2025.

Builds on

31 of 52 references.

References · 31

  1. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  2. 2015Continuous control with deep reinforcement learningcited by 3,279
  3. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problemscited by 1,873
  4. 2020Conservative Q-Learning for Offline Reinforcement Learningcited by 1,777
  5. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learningcited by 1,406
  6. 2021Offline Reinforcement Learning with Implicit Q-Learningcited by 1,245
  7. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,227
  8. 2019Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reductioncited by 833
  9. 2015Dueling Network Architectures for Deep Reinforcement Learningcited by 769
  10. 2020MOPO: Model-based Offline Policy Optimizationcited by 630
  11. 2017Meta-learners for Estimating Heterogeneous Treatment Effects using Machine Learningcited by 475
  12. 2015Doubly Robust Off-policy Value Evaluation for Reinforcement Learningcited by 337
19 more
  1. 2017Quasi-Oracle Estimation of Heterogeneous Treatment Effectscited by 328
  2. 2019Information-Theoretic Considerations in Batch Reinforcement Learningcited by 285
  3. 2021Bellman-consistent Pessimism for Offline Reinforcement Learningcited by 267
  4. 2014A Survey of Multi-Objective Sequential Decision-Makingcited by 229
  5. 2021Online Robust Reinforcement Learning with Model Uncertaintycited by 95
  6. 2022Robust Reinforcement Learning using Offline Datacited by 95
  7. 2021Learning Policies with Zero or Bounded Constraint Violation for Constrained MDPscited by 78
  8. 2023Natural Actor-Critic for Robust Reinforcement Learning with Function Approximationcited by 35
  9. 2023Explicit Feature Interaction-aware Uplift Network for Online Marketingcited by 25
  10. 2021Policy Optimization for Constrained MDPs with Provable Fast Global Convergencecited by 22
  11. 2022Efficient Policy Iteration for Robust Markov Decision Processes via Regularizationcited by 19
  12. 2021Cross DQN: Cross Deep Q Network for Ads Allocation in Feedcited by 18
  13. 2022Anchor-Changing Regularized Natural Policy Gradient for Multi-Objective Reinforcement Learningcited by 15
  14. 2023Ad-load Balancing via Off-policy Learning in a Content Marketplacecited by 11
  15. 2022ResAct: Reinforcing Long-term Engagement in Sequential Recommendation with Residual Actorcited by 10
  16. 2022Deep Page-Level Interest Network in Reinforcement Learning for Ads Allocationcited by 8
  17. 2022Hierarchically Constrained Adaptive Ad Exposure in Feedscited by 6
  18. 2022Hybrid Transfer in Deep Reinforcement Learning for Ads Allocationcited by 5
  19. 2021Blending Advertising with Organic Content in E-Commerce: A Virtual Bids Optimization Approachcited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 1; 2026: 1.

2 papers.

  1. 2025Multi-task Offline Reinforcement Learning for Online Advertising in Recommender Systemscited by 2
  2. 2026Generative Optimization for Incentivized Advertising with Global Level Constraintscited by 0