# Real-Time Bidding by Reinforcement Learning in Display Advertising

arXiv: https://arxiv.org/abs/1701.02490 · HTML: https://mlsanity.com/paper/1701.02490/

## Key facts

- Cited by 91 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 10 Jan 2017.

## Built on by (most cited first, 30 of 91)

- [TorchRL: A data-driven decision-making library for PyTorch](https://mlsanity.com/paper/2306.00577/) (2023)
- [Sustainable Online Reinforcement Learning for Auto-bidding](https://mlsanity.com/paper/2210.07006/) (2022)
- [Generative Auto-Bidding with Value-Guided Explorations](https://mlsanity.com/paper/2504.14587/) (2025)
- [A Cooperative-Competitive Multi-Agent Framework for Auto-bidding in Online Advertising](https://mlsanity.com/paper/2106.06224/) (2021)
- [A Survey on Diffusion Models for Recommender Systems](https://mlsanity.com/paper/2409.05033/) (2024)
- [On the Foundation of Distributionally Robust Reinforcement Learning](https://mlsanity.com/paper/2311.09018/) (2023)
- [Accelerating Offline Reinforcement Learning Application in Real-Time Bidding and Recommendation: Potential Use of Simulation](https://mlsanity.com/paper/2109.08331/) (2021)
- [An Efficient Deep Distribution Network for Bid Shading in First-Price Auctions](https://mlsanity.com/paper/2107.06650/) (2021)
- [RTBAgent: A LLM-based Agent System for Real-Time Bidding](https://mlsanity.com/paper/2502.00792/) (2025)
- [ROI-Constrained Bidding via Curriculum-Guided Bayesian Reinforcement Learning](https://mlsanity.com/paper/2206.05240/) (2022)
- [Ad-load Balancing via Off-policy Learning in a Content Marketplace](https://mlsanity.com/paper/2309.11518/) (2023)
- [Trajectory-wise Iterative Reinforcement Learning Framework for Auto-bidding](https://mlsanity.com/paper/2402.15102/) (2024)
- [Offline Reinforcement Learning for Optimizing Production Bidding Policies](https://mlsanity.com/paper/2310.09426/) (2023)
- [Multi-Agent Cooperative Bidding Games for Multi-Objective Optimization in e-Commercial Sponsored Search](https://mlsanity.com/paper/2106.04075/) (2021)
- [On the Power of Pre-training for Generalization in RL: Provable Benefits and Hardness](https://mlsanity.com/paper/2210.10464/) (2022)
- [Adversarial Constrained Bidding via Minimax Regret Optimization with Causality-Aware Reinforcement Learning](https://mlsanity.com/paper/2306.07106/) (2023)
- [Non-stationary Reinforcement Learning under General Function Approximation](https://mlsanity.com/paper/2306.00861/) (2023)
- [We Know What You Want: An Advertising Strategy Recommender System for Online Advertising](https://mlsanity.com/paper/2105.14188/) (2021)
- [An Empirical Study on Google Research Football Multi-agent Scenarios](https://mlsanity.com/paper/2305.09458/) (2023)
- [An Adaptable Budget Planner for Enhancing Budget-Constrained Auto-Bidding in Online Advertising](https://mlsanity.com/paper/2502.05187/) (2025)
- [See Beyond a Single View: Multi-Attribution Learning Leads to Better Conversion Rate Prediction](https://mlsanity.com/paper/2508.15217/) (2025)
- [Generative Auto-Bidding in Large-Scale Competitive Auctions via Diffusion Completer-Aligner](https://mlsanity.com/paper/2509.03348/) (2025)
- [Real-time Bidding Strategy in Display Advertising: An Empirical Analysis](https://mlsanity.com/paper/2212.02222/) (2022)
- [Neural Insights for Digital Marketing Content Design](https://mlsanity.com/paper/2302.01416/) (2023)
- [Reinforcement Learning with Non-Cumulative Objective](https://mlsanity.com/paper/2307.04957/) (2023)
- [Low-Rank MDPs with Continuous Action Spaces](https://mlsanity.com/paper/2311.03564/) (2023)
- [HiBid: A Cross-Channel Constrained Bidding System with Budget Allocation by Hierarchical Offline Deep Reinforcement Learning](https://mlsanity.com/paper/2312.17503/) (2023)
- [BCRLSP: An Offline Reinforcement Learning Framework for Sequential Targeted Promotion](https://mlsanity.com/paper/2207.07790/) (2022)
- [Insurance pricing on price comparison websites via reinforcement learning](https://mlsanity.com/paper/2308.06935/) (2023)
- [Multi-Session Budget Optimization for Forward Auction-based Federated Learning](https://mlsanity.com/paper/2311.12548/) (2023)

## Abstract (arXiv)

The majority of online display ads are served through real-time bidding (RTB) --- each ad display impression is auctioned off in real-time when it is just being generated from a user visit. To place an ad automatically and optimally, it is critical for advertisers to devise a learning algorithm to cleverly bid an ad impression in real-time. Most previous works consider the bid decision as a static optimization problem of either treating the value of each impression independently or setting a bid price to each segment of ad volume. However, the bidding for a given ad campaign would repeatedly happen during its life span before the budget runs out. As such, each bid is strategically correlated by the constrained budget and the overall effectiveness of the campaign (e.g., the rewards from generated clicks), which is only observed after the campaign has completed. Thus, it is of great interest to devise an optimal bidding strategy sequentially so that the campaign budget can be dynamically allocated across all the available impressions on the basis of both the immediate and future rewards. In this paper, we formulate the bid decision process as a reinforcement learning problem, where the state space is represented by the auction information and the campaign's real-time parameters, while an action is the bid price to set. By modeling the state transition via auction competition, we build a Markov Decision Process framework for learning the optimal bidding policy to optimize the advertising performance in the dynamic real-time bidding environment. Furthermore, the scalability problem from the large real-world auction volume and campaign budget is well handled by state value approximation using neural networks.