# Adversarial Constrained Bidding via Minimax Regret Optimization with Causality-Aware Reinforcement Learning

arXiv: https://arxiv.org/abs/2306.07106 · HTML: https://mlsanity.com/paper/2306.07106/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 25 of its 49 references that are in the map.
- First posted on arXiv 12 Jun 2023.

## Builds on (most cited first, 25 of 25)

- [Generative Adversarial Networks](https://mlsanity.com/paper/2203.00667/) (2022)
- [Convex optimization](https://mlsanity.com/paper/2106.01946/) (2021)
- [Soft Actor-Critic Algorithms and Applications](https://mlsanity.com/paper/1812.05905/) (2018)
- [Deep Variational Information Bottleneck](https://mlsanity.com/paper/1612.00410/) (2016)
- [The Mechanics of n-Player Differentiable Games](https://mlsanity.com/paper/1802.05642/) (2018)
- [Real-Time Bidding by Reinforcement Learning in Display Advertising](https://mlsanity.com/paper/1701.02490/) (2017)
- [Error Bounds of Imitating Policies and Environments](https://mlsanity.com/paper/2010.11876/) (2020)
- [Distributionally Robust Optimization and Generalization in Kernel Methods](https://mlsanity.com/paper/1905.10943/) (2019)
- [Banach Wasserstein GAN](https://mlsanity.com/paper/1806.06621/) (2018)
- [Deep Reinforcement Learning amidst Lifelong Non-Stationarity](https://mlsanity.com/paper/2006.10701/) (2020)
- [Online learning in repeated auctions](https://mlsanity.com/paper/1511.05720/) (2015)
- [Bid Optimization by Multivariable Control in Display Advertising](https://mlsanity.com/paper/1905.10928/) (2019)
- [Learning to Bid Without Knowing your Value](https://mlsanity.com/paper/1711.01333/) (2017)
- [Learning to Bid Optimally and Efficiently in Adversarial First-price Auctions](https://mlsanity.com/paper/2007.04568/) (2020)
- [Auction learning as a two-player game](https://mlsanity.com/paper/2006.05684/) (2020)
- [Online Allocation and Pricing: Constant Regret via Bellman Inequalities](https://mlsanity.com/paper/1906.06361/) (2019)
- [Learning in repeated auctions](https://mlsanity.com/paper/2011.09365/) (2020)
- [Exploration in Online Advertising Systems with Deep Uncertainty-Aware Learning](https://mlsanity.com/paper/2012.02298/) (2020)
- [ROI-Constrained Bidding via Curriculum-Guided Bayesian Reinforcement Learning](https://mlsanity.com/paper/2206.05240/) (2022)
- [Learning Context-aware Task Reasoning for Efficient Meta-reinforcement Learning](https://mlsanity.com/paper/2003.01373/) (2020)
- [Optimal Real-Time Bidding Strategies](https://mlsanity.com/paper/1511.08409/) (2015)
- [Adversarial learning for revenue-maximizing auctions](https://mlsanity.com/paper/1909.06806/) (2019)
- [Reserve Price Optimization for First Price Auctions](https://mlsanity.com/paper/2006.06519/) (2020)
- [Mechanism Design for Value Maximizers](https://mlsanity.com/paper/1607.04362/) (2016)
- [Reserve Pricing in Repeated Second-Price Auctions with Strategic Bidders](https://mlsanity.com/paper/1906.09331/) (2019)

## Built on by (most cited first, 7 of 7)

- [Pixel Reasoner: Incentivizing Pixel-Space Reasoning with Curiosity-Driven Reinforcement Learning](https://mlsanity.com/paper/2505.15966/) (2025)
- [RenderWorld: World Model with Self-Supervised 3D Label](https://mlsanity.com/paper/2409.11356/) (2024)
- [Reverse-Engineered Reasoning for Open-Ended Generation](https://mlsanity.com/paper/2509.06160/) (2025)
- [To Code or not to Code? Adaptive Tool Integration for Math Language Models via Expectation-Maximization](https://mlsanity.com/paper/2502.00691/) (2025)
- [Offline Reinforcement Learning for Optimizing Production Bidding Policies](https://mlsanity.com/paper/2310.09426/) (2023)
- [Generative Large-Scale Pre-trained Models for Automated Ad Bidding Optimization](https://mlsanity.com/paper/2508.02002/) (2025)
- [Order Acquisition Under Competitive Pressure: A Rapidly Adaptive Reinforcement Learning Approach for Ride-Hailing Subsidy Strategies](https://mlsanity.com/paper/2507.02244/) (2025)

## Abstract (arXiv)

The proliferation of the Internet has led to the emergence of online advertising, driven by the mechanics of online auctions. In these repeated auctions, software agents participate on behalf of aggregated advertisers to optimize for their long-term utility. To fulfill the diverse demands, bidding strategies are employed to optimize advertising objectives subject to different spending constraints. Existing approaches on constrained bidding typically rely on i.i.d. train and test conditions, which contradicts the adversarial nature of online ad markets where different parties possess potentially conflicting objectives. In this regard, we explore the problem of constrained bidding in adversarial bidding environments, which assumes no knowledge about the adversarial factors. Instead of relying on the i.i.d. assumption, our insight is to align the train distribution of environments with the potential test distribution meanwhile minimizing policy regret. Based on this insight, we propose a practical Minimax Regret Optimization (MiRO) approach that interleaves between a teacher finding adversarial environments for tutoring and a learner meta-learning its policy over the given distribution of environments. In addition, we pioneer to incorporate expert demonstrations for learning bidding strategies. Through a causality-aware policy design, we improve upon MiRO by distilling knowledge from the experts. Extensive experiments on both industrial data and synthetic data show that our method, MiRO with Causality-aware reinforcement Learning (MiROCL), outperforms prior methods by over 30%.