Marketing Budget Allocation with Offline Constrained Deep Reinforcement Learning
Not on the map. Why?
Abstract (arXiv)
We study the budget allocation problem in online marketing campaigns that utilize previously collected offline data. We first discuss the long-term effect of optimizing marketing budget allocation decisions in the offline setting. To overcome the challenge, we propose a novel game-theoretic offline value-based reinforcement learning method using mixed policies. The proposed method reduces the need to store infinitely many policies in previous methods to only constantly many policies, which achieves nearly optimal policy efficiency, making it practical and favorable for industrial usage. We further show that this method is guaranteed to converge to the optimal policy, which cannot be achieved by previous value-based reinforcement learning methods for marketing budget allocation. Our experiments on a large-scale marketing campaign with tens-of-millions users and more than one billion budget verify the theoretical results and show that the proposed method outperforms various baseline methods. The proposed method has been successfully deployed to serve all the traffic of this marketing campaign.
Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Builds on 30 of its 44 references that are in the map. First posted on arXiv 6 Sep 2023.
Builds on
30 of 44 references.
References · 30
- 2013Playing Atari with Deep Reinforcement Learningcited by 3,274
- 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,417
- 2015Deep Reinforcement Learning with Double Q-learningcited by 1,745
- 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,241
- 2017Constrained Policy Optimizationcited by 983
- 2017DeepFM: A Factorization-Machine based Neural Network for CTR Predictioncited by 676
- 2017Deep Interest Network for Click-Through Rate Predictioncited by 544
- 2018Reward Constrained Policy Optimizationcited by 417
- 2015Risk-Constrained Reinforcement Learning with Percentile Risk Criteriacited by 337
- 2020Responsive Safety in Reinforcement Learning by PID Lagrangian Methodscited by 316
- 2019Batch Policy Learning under Constraintscited by 244
- 2018Provably Efficient Maximum Entropy Explorationcited by 242
18 more
- 2021Reinforcement learning based recommender systems: A surveycited by 188
- 2019Constrained Reinforcement Learning Has Zero Duality Gapcited by 151
- 2021Reward is enough for convex MDPscited by 84
- 2018Virtual-Taobao: Virtualizing Real-world Online Retail Environment for Reinforcement Learningcited by 80
- 2018Accelerated Primal-Dual Policy Optimization for Safe Reinforcement Learningcited by 62
- 2018Budget Constrained Bidding by Model-free Reinforcement Learning in Display Advertisingcited by 61
- 2019Reinforcement Learning with Convex Constraintscited by 60
- 2021A Primal-Dual Approach to Constrained Markov Decision Processescited by 42
- 2019Value constrained model-free continuous controlcited by 34
- 2019A Unified Framework for Marketing Budget Allocationcited by 33
- 2018Deep Reinforcement Learning based Recommendation with Explicit User-Item Interactions Modelingcited by 31
- 2021A Survey of Deep Reinforcement Learning in Recommender Systems: A Systematic Review and Future Directionscited by 21
- 2023Model-based Constrained MDP for Budget Allocation in Sequential Incentive Marketingcited by 15
- 2015Smart Pacing for Effective Online Ad Campaign Optimizationcited by 14
- 2019Apprenticeship Learning via Frank-Wolfecited by 13
- 2020Graph Representation Learning for Merchant Incentive Optimization in Mobile Payment Marketingcited by 7
- 2021Adversarial Learning for Incentive Optimization in Mobile Payment Marketingcited by 4
- 2021Large-Scale Data-Driven Airline Market Influence Maximizationcited by 2
Built on by
Citing papers per year: 2025: 1; 2026: 2.
3 papers.