# Real-Time Bidding with Multi-Agent Reinforcement Learning in Display Advertising

arXiv: https://arxiv.org/abs/1802.09756 · HTML: https://mlsanity.com/paper/1802.09756/

## Key facts

- Cited by 65 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 27 Feb 2018.

## Built on by (most cited first, 30 of 65)

- [TorchRL: A data-driven decision-making library for PyTorch](https://mlsanity.com/paper/2306.00577/) (2023)
- [Deep Multiagent Reinforcement Learning: Challenges and Directions](https://mlsanity.com/paper/2106.15691/) (2021)
- [Multi-agent Reinforcement Learning: A Comprehensive Survey](https://mlsanity.com/paper/2312.10256/) (2023)
- [Mean-Field Multi-Agent Reinforcement Learning: A Decentralized Network Approach](https://mlsanity.com/paper/2108.02731/) (2021)
- [Optimal-er Auctions through Attention](https://mlsanity.com/paper/2202.13110/) (2022)
- [Policy Gradient Methods Find the Nash Equilibrium in N-player General-sum Linear-quadratic Games](https://mlsanity.com/paper/2107.13090/) (2021)
- [A Cooperative-Competitive Multi-Agent Framework for Auto-bidding in Online Advertising](https://mlsanity.com/paper/2106.06224/) (2021)
- [Optimizing Nitrogen Management with Deep Reinforcement Learning and Crop Simulations](https://mlsanity.com/paper/2204.10394/) (2022)
- [Accelerating Offline Reinforcement Learning Application in Real-Time Bidding and Recommendation: Potential Use of Simulation](https://mlsanity.com/paper/2109.08331/) (2021)
- [Causal Multi-Agent Reinforcement Learning: Review and Open Problems](https://mlsanity.com/paper/2111.06721/) (2021)
- [Distributed TD(0) with Almost No Communication](https://mlsanity.com/paper/2305.16246/) (2023)
- [Flatland Competition 2020: MAPF and MARL for Efficient Train Coordination on a Grid World](https://mlsanity.com/paper/2103.16511/) (2021)
- [Statistically Efficient Advantage Learning for Offline Reinforcement Learning in Infinite Horizons](https://mlsanity.com/paper/2202.13163/) (2022)
- [Mean-Field Sampling for Cooperative Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2412.00661/) (2024)
- [Multi-Agent Cooperative Bidding Games for Multi-Objective Optimization in e-Commercial Sponsored Search](https://mlsanity.com/paper/2106.04075/) (2021)
- [Hierarchically Constrained Adaptive Ad Exposure in Feeds](https://mlsanity.com/paper/2205.15759/) (2022)
- [Exploiting Structure in Offline Multi-Agent RL: The Benefits of Low Interaction Rank](https://mlsanity.com/paper/2410.01101/) (2024)
- [Multi-Agent Reinforcement Learning: Methods, Applications, Visionary Prospects, and Challenges](https://mlsanity.com/paper/2305.10091/) (2023)
- [See Beyond a Single View: Multi-Attribution Learning Leads to Better Conversion Rate Prediction](https://mlsanity.com/paper/2508.15217/) (2025)
- [Generative Auto-Bidding in Large-Scale Competitive Auctions via Diffusion Completer-Aligner](https://mlsanity.com/paper/2509.03348/) (2025)
- [Real-time Bidding Strategy in Display Advertising: An Empirical Analysis](https://mlsanity.com/paper/2212.02222/) (2022)
- [Coordinated Dynamic Bidding in Repeated Second-Price Auctions with Budgets](https://mlsanity.com/paper/2306.07709/) (2023)
- [On Feasible Rewards in Multi-Agent Inverse Reinforcement Learning](https://mlsanity.com/paper/2411.15046/) (2024)
- [Truncation-Free Matching System for Display Advertising at Alibaba](https://mlsanity.com/paper/2102.09283/) (2021)
- [BCRLSP: An Offline Reinforcement Learning Framework for Sequential Targeted Promotion](https://mlsanity.com/paper/2207.07790/) (2022)
- [PlayBest: Professional Basketball Player Behavior Synthesis via Planning with Diffusion](https://mlsanity.com/paper/2306.04090/) (2023)
- [Improve ROI with Causal Learning and Conformal Prediction](https://mlsanity.com/paper/2407.01065/) (2024)
- [Auto-bidding in real-time auctions via Oracle Imitation Learning (OIL)](https://mlsanity.com/paper/2412.11434/) (2024)
- [Generative Large-Scale Pre-trained Models for Automated Ad Bidding Optimization](https://mlsanity.com/paper/2508.02002/) (2025)
- [Impression Allocation and Policy Search in Display Advertising](https://mlsanity.com/paper/2203.07073/) (2022)

## Abstract (arXiv)

Real-time advertising allows advertisers to bid for each impression for a visiting user. To optimize specific goals such as maximizing revenue and return on investment (ROI) led by ad placements, advertisers not only need to estimate the relevance between the ads and user's interests, but most importantly require a strategic response with respect to other advertisers bidding in the market. In this paper, we formulate bidding optimization with multi-agent reinforcement learning. To deal with a large number of advertisers, we propose a clustering method and assign each cluster with a strategic bidding agent. A practical Distributed Coordinated Multi-Agent Bidding (DCMAB) has been proposed and implemented to balance the tradeoff between the competition and cooperation among advertisers. The empirical study on our industry-scaled real-world data has demonstrated the effectiveness of our methods. Our results show cluster-based bidding would largely outperform single-agent and bandit approaches, and the coordinated bidding achieves better overall objectives than purely self-interested bidding agents.