mlsanityData to 14 Oct 2026

Learning Personalized Ad Impact via Contextual Reinforcement Learning under Delayed Rewards

arXiv 2510.20055 ↗First posted 22 Oct 2025cs.LG stat.MLcited by 0

On the map: Deep reinforcement learning methods → RL theory and sample complexity

Abstract (arXiv)

Online advertising platforms use automated auctions to connect advertisers with potential customers, requiring effective bidding strategies to maximize profits. Accurate ad impact estimation requires considering three key factors: delayed and long-term effects, cumulative ad impacts such as reinforcement or fatigue, and customer heterogeneity. However, these effects are often not jointly addressed in previous studies. To capture these factors, we model ad bidding as a Contextual Markov Decision Process (CMDP) with delayed Poisson rewards. For efficient estimation, we propose a two-stage maximum likelihood estimator combined with data-splitting strategies, ensuring controlled estimation error based on the first-stage estimator's (in)accuracy. Building on this, we design a reinforcement learning algorithm to derive efficient personalized bidding strategies. This approach achieves a near-optimal regret bound of tildeO(dH2√T), where d is the contextual dimension, H is the number of rounds, and T is the number of customers. Our theoretical findings are validated by simulation experiments.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in RL theory and sample complexity, one of 15 directions in Deep reinforcement learning methods. Builds on 21 of its 53 references that are in the map. First posted on arXiv 22 Oct 2025.

Builds on

21 of 53 references.

References · 21

  1. 2019Provably Efficient Reinforcement Learning with Linear Function Approximationcited by 520
  2. 2015Contextual Markov Decision Processescited by 220
  3. 2022Nearly Optimal Algorithms for Linear Contextual Bandits with Adversarial Corruptionscited by 51
  4. 2015Online learning in repeated auctionscited by 39
  5. 2017Learning to Bid Without Knowing your Valuecited by 34
  6. 2020Optimal No-regret Learning in Repeated First-price Auctionscited by 33
  7. 2024Auto-bidding and Auctions in Online Advertising: A Surveycited by 26
  8. 2020Online Sparse Reinforcement Learningcited by 25
  9. 2018Bidding Machine: Learning to Bid for Directly Optimizing Profits in Display Advertisingcited by 21
  10. 2021Learning to Bid in Contextual First Price Auctionscited by 19
  11. 2022Learning Neural Contextual Bandits Through Perturbed Rewardscited by 17
  12. 2022Optimism in Face of a Context: Regret Guarantees for Stochastic Contextual MDPcited by 11
9 more
  1. 2022Leveraging the Hints: Adaptive Bidding in Repeated First-Price Auctionscited by 11
  2. 2023Efficient Algorithms for Generalized Linear Bandits with Heavy-tailed Rewardscited by 11
  3. 2022Learning Efficiently Function Approximation for Contextual MDPcited by 10
  4. 2023Efficient Rate Optimal Regret for Adversarial Contextual MDPs Using Online Function Approximationcited by 9
  5. 2019Online Second Price Auction with Semi-bandit Feedback Under the Non-Stationary Settingcited by 6
  6. 2022Incrementality Bidding via Reinforcement Learning under Mixed and Delayed Rewardscited by 5
  7. 2023Online Learning in Contextual Second-Price Pay-Per-Click Auctionscited by 4
  8. 2019Estimating Individual Advertising Effect in E-Commercecited by 3
  9. 2024Improved Online Learning Algorithms for CTR Prediction in Ad Auctionscited by 2

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2024AIGB: Generative Auto-bidding via Conditional Diffusion Modeling
  2. 2026Learning to Allocate Incentives for Incentivized Advertising via Offline Model-Based Reinforcement Learning
  3. 2024Optimizing Search Advertising Strategies: Integrating Reinforcement Learning with Generalized Second-Price Auctions for Enhanced Ad Ranking and Bidding
  4. 2025Generative Large-Scale Pre-trained Models for Automated Ad Bidding Optimization
  5. 2026AHBid: An Adaptable Hierarchical Bidding Framework for Cross-Channel Advertising
  6. 2025Multi-task Offline Reinforcement Learning for Online Advertising in Recommender Systems
  7. 2026DARA: Few-shot Budget Allocation in Online Advertising via In-Context Decision Making with RL-Finetuned LLMs
  8. 2024Auto-bidding in real-time auctions via Oracle Imitation Learning (OIL)