# Deep Reinforcement Learning for Sponsored Search Real-time Bidding

arXiv: https://arxiv.org/abs/1803.00259 · HTML: https://mlsanity.com/paper/1803.00259/

## Key facts

- Cited by 34 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 1 Mar 2018.

## Built on by (most cited first, 30 of 34)

- [Generative Auto-Bidding with Value-Guided Explorations](https://mlsanity.com/paper/2504.14587/) (2025)
- [Intelligent Electric Vehicle Charging Recommendation Based on Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2102.07359/) (2021)
- [On the Foundation of Distributionally Robust Reinforcement Learning](https://mlsanity.com/paper/2311.09018/) (2023)
- [Accelerating Offline Reinforcement Learning Application in Real-Time Bidding and Recommendation: Potential Use of Simulation](https://mlsanity.com/paper/2109.08331/) (2021)
- [RTBAgent: A LLM-based Agent System for Real-Time Bidding](https://mlsanity.com/paper/2502.00792/) (2025)
- [A Survey on Deep Reinforcement Learning for Data Processing and Analytics](https://mlsanity.com/paper/2108.04526/) (2021)
- [Offline Reinforcement Learning for Optimizing Production Bidding Policies](https://mlsanity.com/paper/2310.09426/) (2023)
- [Multi-Agent Cooperative Bidding Games for Multi-Objective Optimization in e-Commercial Sponsored Search](https://mlsanity.com/paper/2106.04075/) (2021)
- [An Adaptable Budget Planner for Enhancing Budget-Constrained Auto-Bidding in Online Advertising](https://mlsanity.com/paper/2502.05187/) (2025)
- [Generative Auto-Bidding in Large-Scale Competitive Auctions via Diffusion Completer-Aligner](https://mlsanity.com/paper/2509.03348/) (2025)
- [TSI: an Ad Text Strength Indicator using Text-to-CTR and Semantic-Ad-Similarity](https://mlsanity.com/paper/2108.08226/) (2021)
- [HiBid: A Cross-Channel Constrained Bidding System with Budget Allocation by Hierarchical Offline Deep Reinforcement Learning](https://mlsanity.com/paper/2312.17503/) (2023)
- [BCRLSP: An Offline Reinforcement Learning Framework for Sequential Targeted Promotion](https://mlsanity.com/paper/2207.07790/) (2022)
- [Auto-bidding in real-time auctions via Oracle Imitation Learning (OIL)](https://mlsanity.com/paper/2412.11434/) (2024)
- [Think Before Writing: Feature-Level Multi-Objective Optimization for Generative Citation Visibility](https://mlsanity.com/paper/2604.19113/) (2026)
- [AdCraft: An Advanced Reinforcement Learning Benchmark Environment for Search Engine Marketing Optimization](https://mlsanity.com/paper/2306.11971/) (2023)
- [Online Bidding under RoS Constraints without Knowing the Value](https://mlsanity.com/paper/2503.03195/) (2025)
- [Multi-task Offline Reinforcement Learning for Online Advertising in Recommender Systems](https://mlsanity.com/paper/2506.23090/) (2025)
- [Bid Optimization using Maximum Entropy Reinforcement Learning](https://mlsanity.com/paper/2110.05032/) (2021)
- [A Reinforcement Learning Approach in Multi-Phase Second-Price Auction Design](https://mlsanity.com/paper/2210.10278/) (2022)
- [Deep Reinforcement Learning for Ranking Utility Tuning in the Ad Recommender System at Pinterest](https://mlsanity.com/paper/2509.05292/) (2025)
- [Autobidding Arena: unified evaluation of the classical and RL-based autobidding algorithms](https://mlsanity.com/paper/2510.19357/) (2025)
- [C2:Cross learning module enhanced decision transformer with Constraint-aware loss for auto-bidding](https://mlsanity.com/paper/2601.20257/) (2026)
- [PRO-Bid: Pareto-Prioritized Regret Optimization for Constraint-Aware Generative Auto-Bidding](https://mlsanity.com/paper/2602.08261/) (2026)
- [Techniques Toward Optimizing Viewability in RTB Ad Campaigns Using Reinforcement Learning](https://mlsanity.com/paper/2105.10587/) (2021)
- [SuperCone: Unified User Segmentation over Heterogeneous Experts via Concept Meta-learning](https://mlsanity.com/paper/2203.07029/) (2022)
- [Risk-Aware Bid Optimization for Online Display Advertisement](https://mlsanity.com/paper/2210.15837/) (2022)
- [Bidding-Aware Retrieval for Multi-Stage Consistency in Online Advertising](https://mlsanity.com/paper/2508.05206/) (2025)
- [Q-Regularized Generative Auto-Bidding: From Suboptimal Trajectories to Optimal Policies](https://mlsanity.com/paper/2601.02754/) (2026)
- [AHBid: An Adaptable Hierarchical Bidding Framework for Cross-Channel Advertising](https://mlsanity.com/paper/2602.22650/) (2026)

## Abstract (arXiv)

Bidding optimization is one of the most critical problems in online advertising. Sponsored search (SS) auction, due to the randomness of user query behavior and platform nature, usually adopts keyword-level bidding strategies. In contrast, the display advertising (DA), as a relatively simpler scenario for auction, has taken advantage of real-time bidding (RTB) to boost the performance for advertisers. In this paper, we consider the RTB problem in sponsored search auction, named SS-RTB. SS-RTB has a much more complex dynamic environment, due to stochastic user query behavior and more complex bidding policies based on multiple keywords of an ad. Most previous methods for DA cannot be applied. We propose a reinforcement learning (RL) solution for handling the complex dynamic environment. Although some RL methods have been proposed for online advertising, they all fail to address the "environment changing" problem: the state transition probabilities vary between two days. Motivated by the observation that auction sequences of two days share similar transition patterns at a proper aggregation level, we formulate a robust MDP model at hour-aggregation level of the auction data and propose a control-by-model framework for SS-RTB. Rather than generating bid prices directly, we decide a bidding model for impressions of each hour and perform real-time bidding accordingly. We also extend the method to handle the multi-agent problem. We deployed the SS-RTB system in the e-commerce search auction platform of Alibaba. Empirical experiments of offline evaluation and online A/B test demonstrate the effectiveness of our method.