# A Cooperative-Competitive Multi-Agent Framework for Auto-bidding in Online Advertising

arXiv: https://arxiv.org/abs/2106.06224 · HTML: https://mlsanity.com/paper/2106.06224/

## Key facts

- Cited by 22 papers in the map (data through 14 Oct 2026).
- Builds on 19 of its 39 references that are in the map.
- First posted on arXiv 11 Jun 2021.

## Builds on (most cited first, 19 of 19)

- [High-Dimensional Continuous Control Using Generalized Advantage Estimation](https://mlsanity.com/paper/1506.02438/) (2015)
- [Multi-Agent Actor-Critic for Mixed Cooperative-Competitive Environments](https://mlsanity.com/paper/1706.02275/) (2017)
- [Counterfactual Multi-Agent Policy Gradients](https://mlsanity.com/paper/1705.08926/) (2017)
- [QMIX: Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/1803.11485/) (2018)
- [Value-Decomposition Networks For Cooperative Multi-Agent Learning](https://mlsanity.com/paper/1706.05296/) (2017)
- [Multi-agent Reinforcement Learning in Sequential Social Dilemmas](https://mlsanity.com/paper/1702.03037/) (2017)
- [Mean Field Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/1802.05438/) (2018)
- [Multiagent Cooperation and Competition with Deep Reinforcement Learning](https://mlsanity.com/paper/1511.08779/) (2015)
- [A Survey and Critique of Multiagent Deep Reinforcement Learning](https://mlsanity.com/paper/1810.05587/) (2018)
- [Real-Time Bidding by Reinforcement Learning in Display Advertising](https://mlsanity.com/paper/1701.02490/) (2017)
- [Real-Time Bidding with Multi-Agent Reinforcement Learning in Display Advertising](https://mlsanity.com/paper/1802.09756/) (2018)
- [Budget Constrained Bidding by Model-free Reinforcement Learning in Display Advertising](https://mlsanity.com/paper/1802.08365/) (2018)
- [Display Advertising with Real-Time Bidding (RTB) and Behavioural Targeting](https://mlsanity.com/paper/1610.03013/) (2016)
- [Neural Auction: End-to-End Learning of Auction Mechanisms for E-Commerce Advertising](https://mlsanity.com/paper/2106.03593/) (2021)
- [Learning Theory and Algorithms for Revenue Optimization in Second-Price Auctions with Reserve](https://mlsanity.com/paper/1310.5665/) (2013)
- [SMIX(λ): Enhancing Centralized Value Functions for Cooperative Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/1911.04094/) (2019)
- [Reinforcement Mechanism Design, with Applications to Dynamic Pricing in Sponsored Search Auctions](https://mlsanity.com/paper/1711.10279/) (2017)
- [AiAds: Automated and Intelligent Advertising System for Sponsored Search](https://mlsanity.com/paper/1907.12118/) (2019)
- [Multi-Agent Cooperative Bidding Games for Multi-Objective Optimization in e-Commercial Sponsored Search](https://mlsanity.com/paper/2106.04075/) (2021)

## Built on by (most cited first, 22 of 22)

- [Generative Auto-Bidding with Value-Guided Explorations](https://mlsanity.com/paper/2504.14587/) (2025)
- [GAS: Generative Auto-bidding with Post-training Search](https://mlsanity.com/paper/2412.17018/) (2024)
- [Generative Auto-Bidding in Large-Scale Competitive Auctions via Diffusion Completer-Aligner](https://mlsanity.com/paper/2509.03348/) (2025)
- [Real-time Bidding Strategy in Display Advertising: An Empirical Analysis](https://mlsanity.com/paper/2212.02222/) (2022)
- [Coordinated Dynamic Bidding in Repeated Second-Price Auctions with Budgets](https://mlsanity.com/paper/2306.07709/) (2023)
- [Auto-bidding in real-time auctions via Oracle Imitation Learning (OIL)](https://mlsanity.com/paper/2412.11434/) (2024)
- [Generative Large-Scale Pre-trained Models for Automated Ad Bidding Optimization](https://mlsanity.com/paper/2508.02002/) (2025)
- [Expert-Guided Diffusion Planner for Auto-Bidding](https://mlsanity.com/paper/2508.08687/) (2025)
- [Lightweight Auto-bidding based on Traffic Prediction in Live Advertising](https://mlsanity.com/paper/2508.06069/) (2025)
- [Enhancing Generative Auto-bidding with Offline Reward Evaluation and Policy Search](https://mlsanity.com/paper/2509.15927/) (2025)
- [Autobidding Arena: unified evaluation of the classical and RL-based autobidding algorithms](https://mlsanity.com/paper/2510.19357/) (2025)
- [Bid2X: Revealing Dynamics of Bidding Environment in Online Advertising from A Foundation Model Lens](https://mlsanity.com/paper/2510.23410/) (2025)
- [PlatformBid: An Auto-Bidding Benchmark from a Unified Advertising Platform's Perspective](https://mlsanity.com/paper/2607.27265/) (2026)
- [Bandits for Sponsored Search Auctions under Unknown Valuation Model: Case Study in E-Commerce Advertising](https://mlsanity.com/paper/2304.00999/) (2023)
- [Hierarchical Multi-agent Meta-Reinforcement Learning for Cross-channel Bidding](https://mlsanity.com/paper/2412.19064/) (2024)
- [Large-Scale Auto-bidding with Nash Equilibrium Constraints](https://mlsanity.com/paper/2503.10304/) (2025)
- [Q-Regularized Generative Auto-Bidding: From Suboptimal Trajectories to Optimal Policies](https://mlsanity.com/paper/2601.02754/) (2026)
- [Knowledge-informed Bidding with Dual-process Control for Online Advertising](https://mlsanity.com/paper/2603.04920/) (2026)
- [Generative Auto-Bidding with Unified Modeling and Exploration](https://mlsanity.com/paper/2605.19457/) (2026)
- [Constrained Auto-Bidding via Generative Response Modeling](https://mlsanity.com/paper/2605.27811/) (2026)
- [AIGB-R1: Self-Evolving Generative Auto-Bidding via Hierarchical Planner-Executor Optimization](https://mlsanity.com/paper/2607.17281/) (2026)
- [OneBid: A Unified Auto-Bidding Foundation Model for Diverse oCPX Advertising Scenarios](https://mlsanity.com/paper/2609.21550/) (2026)

## Abstract (arXiv)

In online advertising, auto-bidding has become an essential tool for advertisers to optimize their preferred ad performance metrics by simply expressing high-level campaign objectives and constraints. Previous works designed auto-bidding tools from the view of single-agent, without modeling the mutual influence between agents. In this paper, we instead consider this problem from a distributed multi-agent perspective, and propose a general underlineMulti-underlineAgent reinforcement learning framework for underlineAuto-underlineBidding, namely MAAB, to learn the auto-bidding strategies. First, we investigate the competition and cooperation relation among auto-bidding agents, and propose a temperature-regularized credit assignment to establish a mixed cooperative-competitive paradigm. By carefully making a competition and cooperation trade-off among agents, we can reach an equilibrium state that guarantees not only individual advertiser's utility but also the system performance (i.e., social welfare). Second, to avoid the potential collusion behaviors of bidding low prices underlying the cooperation, we further propose bar agents to set a personalized bidding bar for each agent, and then alleviate the revenue degradation due to the cooperation. Third, to deploy MAAB in the large-scale advertising system with millions of advertisers, we propose a mean-field approach. By grouping advertisers with the same objective as a mean auto-bidding agent, the interactions among the large-scale advertisers are greatly simplified, making it practical to train MAAB efficiently. Extensive experiments on the offline industrial dataset and Alibaba advertising platform demonstrate that our approach outperforms several baseline methods in terms of social welfare and revenue.