mlsanityData to 14 Oct 2026

HiBid: A Cross-Channel Constrained Bidding System with Budget Allocation by Hierarchical Offline Deep Reinforcement Learning

arXiv 2312.17503 ↗First posted 29 Dec 2023cs.LG cs.GTcited by 4

On the map: Offline RL and imitation learning from data → Decision transformers and sequence modeling

Abstract (arXiv)

Online display advertising platforms service numerous advertisers by providing real-time bidding (RTB) for the scale of billions of ad requests every day. The bidding strategy handles ad requests cross multiple channels to maximize the number of clicks under the set financial constraints, i.e., total budget and cost-per-click (CPC), etc. Different from existing works mainly focusing on single channel bidding, we explicitly consider cross-channel constrained bidding with budget allocation. Specifically, we propose a hierarchical offline deep reinforcement learning (DRL) framework called ``HiBid'', consisted of a high-level planner equipped with auxiliary loss for non-competitive budget allocation, and a data augmentation enhanced low-level executor for adaptive bidding strategy in response to allocated budgets. Additionally, a CPC-guided action selection mechanism is introduced to satisfy the cross-channel CPC constraint. Through extensive experiments on both the large-scale log data and online A/B testing, we confirm that HiBid outperforms six baselines in terms of the number of clicks, CPC satisfactory ratio, and return-on-investment (ROI). We also deploy HiBid on Meituan advertising platform to already service tens of thousands of advertisers every day.

Key facts. Cited by 4 papers in the map (data through 14 Oct 2026). Placed in Decision transformers and sequence modeling, one of 13 directions in Offline RL and imitation learning from data. Builds on 19 of its 47 references that are in the map. First posted on arXiv 29 Dec 2023.

Builds on

19 of 47 references.

References · 19

  1. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,241
  2. 2019Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reductioncited by 840
  3. 2018Reward Constrained Policy Optimizationcited by 417
  4. 2019Way Off-Policy Batch Deep Reinforcement Learning of Implicit Human Preferences in Dialogcited by 284
  5. 2019Batch Policy Learning under Constraintscited by 244
  6. 2019Constrained Reinforcement Learning Has Zero Duality Gapcited by 151
  7. 2022Mildly Conservative Q-Learning for Offline Reinforcement Learningcited by 112
  8. 2017Real-Time Bidding by Reinforcement Learning in Display Advertisingcited by 91
  9. 2018Budget Constrained Bidding by Model-free Reinforcement Learning in Display Advertisingcited by 61
  10. 2017Optimized Cost per Click in Taobao Display Advertisingcited by 57
  11. 2021Hierarchical Skills for Efficient Explorationcited by 38
  12. 2019Bid Optimization by Multivariable Control in Display Advertisingcited by 37
7 more
  1. 2018Deep Reinforcement Learning for Sponsored Search Real-time Biddingcited by 34
  2. 2023Policy Regularization with Dataset Constraint for Offline Reinforcement Learningcited by 34
  3. 2018Bidding Machine: Learning to Bid for Directly Optimizing Profits in Display Advertisingcited by 21
  4. 2021Cross DQN: Cross Deep Q Network for Ads Allocation in Feedcited by 19
  5. 2023Model-based Constrained MDP for Budget Allocation in Sequential Incentive Marketingcited by 15
  6. 2021An Efficient Deep Distribution Network for Bid Shading in First-Price Auctionscited by 14
  7. 2022ROI-Constrained Bidding via Curriculum-Guided Bayesian Reinforcement Learningcited by 12

Built on by

Cited by year

Citing papers per year: 2024: 1; 2025: 3.

4 papers.

  1. 2025An Adaptable Budget Planner for Enhancing Budget-Constrained Auto-Bidding in Online Advertisingcited by 5
  2. 2025Reinforcement Learning for Game-Theoretic Resource Allocation on Graphscited by 1
  3. 2025Enhancing Generative Auto-bidding with Offline Reward Evaluation and Policy Searchcited by 1
  4. 2024Hierarchical Multi-agent Meta-Reinforcement Learning for Cross-channel Biddingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026AHBid: An Adaptable Hierarchical Bidding Framework for Cross-Channel Advertising
  2. 2023Offline Reinforcement Learning for Optimizing Production Bidding Policies
  3. 2026HOBA: Hierarchical On-Policy Bidding Agents for Adaptive Online Advertising
  4. 2026OneBid: A Unified Auto-Bidding Foundation Model for Diverse oCPX Advertising Scenarios
  5. 2026DARA: Few-shot Budget Allocation in Online Advertising via In-Context Decision Making with RL-Finetuned LLMs
  6. 2026Learning to Allocate Incentives for Incentivized Advertising via Offline Model-Based Reinforcement Learning
  7. 2020Optimal Bidding Strategy without Exploration in Real-time Bidding
  8. 2026DRIVE: Distributional and Retrieval-Augmented Bidding with Value Evaluation