mlsanityData to 7 Oct 2026

ROI-Constrained Bidding via Curriculum-Guided Bayesian Reinforcement Learning

arXiv 2206.05240 ↗First posted 10 Jun 2022cs.LG cs.AIcited by 12

Not on the map. Why?

Abstract (arXiv)

Real-Time Bidding (RTB) is an important mechanism in modern online advertising systems. Advertisers employ bidding strategies in RTB to optimize their advertising effects subject to various financial requirements, especially the return-on-investment (ROI) constraint. ROIs change non-monotonically during the sequential bidding process, and often induce a see-saw effect between constraint satisfaction and objective optimization. While some existing approaches show promising results in static or mildly changing ad markets, they fail to generalize to highly dynamic ad markets with ROI constraints, due to their inability to adaptively balance constraints and objectives amidst non-stationarity and partial observability. In this work, we specialize in ROI-Constrained Bidding in non-stationary markets. Based on a Partially Observable Constrained Markov Decision Process, our method exploits an indicator-augmented reward function free of extra trade-off parameters and develops a Curriculum-Guided Bayesian Reinforcement Learning (CBRL) framework to adaptively control the constraint-objective trade-off in non-stationary ad markets. Extensive experiments on a large-scale industrial dataset with two problem settings reveal that CBRL generalizes well in both in-distribution and out-of-distribution data regimes, and enjoys superior learning efficiency and stability.

Key facts. Cited by 12 papers in the map (data through 7 Oct 2026). Builds on 22 of its 44 references that are in the map. First posted on arXiv 10 Jun 2022.

Builds on

22 of 44 references.

References · 22

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  3. 2013Auto-Encoding Variational Bayescited by 3,980
  4. 2015Continuous control with deep reinforcement learningcited by 3,279
  5. 2013Playing Atari with Deep Reinforcement Learningcited by 3,211
  6. 2016Variational Inference: A Review for Statisticianscited by 1,466
  7. 2017Curiosity-driven Exploration by Self-supervised Predictioncited by 1,173
  8. 2017Constrained Policy Optimizationcited by 928
  9. 2018Reward Constrained Policy Optimizationcited by 397
  10. 2015Risk-Constrained Reinforcement Learning with Percentile Risk Criteriacited by 320
  11. 2013(More) Efficient Reinforcement Learning via Posterior Samplingcited by 233
  12. 2019VariBAD: A Very Good Method for Bayes-Adaptive Deep RL via Meta-Learningcited by 210
10 more
  1. 2019Lyapunov-based Safe Policy Optimization for Continuous Controlcited by 140
  2. 2018Deep Variational Reinforcement Learning for POMDPscited by 122
  3. 2017Real-Time Bidding by Reinforcement Learning in Display Advertisingcited by 90
  4. 2019Meta reinforcement learning as task inferencecited by 75
  5. 2018Budget Constrained Bidding by Model-free Reinforcement Learning in Display Advertisingcited by 60
  6. 2019Bid Optimization by Multivariable Control in Display Advertisingcited by 37
  7. 2013Real-time Bidding for Online Advertising: Measurement and Analysiscited by 32
  8. 2020Exploration in Online Advertising Systems with Deep Uncertainty-Aware Learningcited by 12
  9. 2020Learning Context-aware Task Reasoning for Efficient Meta-reinforcement Learningcited by 11
  10. 2016Mechanism Design for Value Maximizerscited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 2; 2024: 2; 2025: 7; 2026: 1.

12 papers.

  1. 2025VL-Rethinker: Incentivizing Self-Reflection of Vision-Language Models with Reinforcement Learningcited by 315
  2. 2025Pixel Reasoner: Incentivizing Pixel-Space Reasoning with Curiosity-Driven Reinforcement Learningcited by 267
  3. 2025VR-Thinker: Boosting Video Reward Models through Thinking-with-Image Reasoningcited by 11
  4. 2023Adversarial Constrained Bidding via Minimax Regret Optimization with Causality-Aware Reinforcement Learningcited by 7
  5. 2025Generative Auto-Bidding in Large-Scale Competitive Auctions via Diffusion Completer-Alignercited by 5
  6. 2023HiBid: A Cross-Channel Constrained Bidding System with Budget Allocation by Hierarchical Offline Deep Reinforcement Learningcited by 4
  7. 2024Auto-bidding in real-time auctions via Oracle Imitation Learning (OIL)cited by 3
  8. 2025Autobidding Arena: unified evaluation of the classical and RL-based autobidding algorithmscited by 1
  9. 2024Hierarchical Multi-agent Meta-Reinforcement Learning for Cross-channel Biddingcited by 0
  10. 2025Order Acquisition Under Competitive Pressure: A Rapidly Adaptive Reinforcement Learning Approach for Ride-Hailing Subsidy Strategiescited by 0
  11. 2025HALO: Hindsight-Augmented Learning for Online Auto-Biddingcited by 0
  12. 2026A Lightweight MPC Bidding Framework for Brand Auction Adscited by 0