# Deep Reinforcement Learning for Sequential Combinatorial Auctions

arXiv: https://arxiv.org/abs/2407.08022 · HTML: https://mlsanity.com/paper/2407.08022/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Placed in Adversarial robustness and RL in finance, one of 15 directions in Deep reinforcement learning methods.
- Builds on 25 of its 37 references that are in the map.
- First posted on arXiv 10 Jul 2024.

## Placed in

- Deep reinforcement learning methods / [Adversarial robustness and RL in finance](https://mlsanity.com/deeprl/adversarial-robustness-and-rl-in-finance/)

## Builds on (most cited first, 25 of 25)

- [Differentiable Convex Optimization Layers](https://mlsanity.com/paper/1910.12430/) (2019)
- [DiffTaichi: Differentiable Programming for Physical Simulation](https://mlsanity.com/paper/1910.00935/) (2019)
- [A Differentiable Programming System to Bridge Machine Learning and Scientific Computing](https://mlsanity.com/paper/1907.07587/) (2019)
- [Scalable Differentiable Physics for Learning and Control](https://mlsanity.com/paper/2007.02168/) (2020)
- [Optimal Auctions through Deep Learning: Advances in Differentiable Economics](https://mlsanity.com/paper/1706.03459/) (2017)
- [Automated Mechanism Design via Neural Networks](https://mlsanity.com/paper/1805.03382/) (2018)
- [A Permutation-Equivariant Neural Network Architecture For Auction Design](https://mlsanity.com/paper/2003.01497/) (2020)
- [Auction learning as a two-player game](https://mlsanity.com/paper/2006.05684/) (2020)
- [Optimal-er Auctions through Attention](https://mlsanity.com/paper/2202.13110/) (2022)
- [A Scalable Neural Network for DSIC Affine Maximizer Auction Design](https://mlsanity.com/paper/2305.12162/) (2023)
- [PreferenceNet: Encoding Human Preferences in Auction Design with Deep Learning](https://mlsanity.com/paper/2106.03215/) (2021)
- [Training Efficient Controllers via Analytic Policy Gradient](https://mlsanity.com/paper/2209.13052/) (2022)
- [Certifying Strategyproof Auction Networks](https://mlsanity.com/paper/2006.08742/) (2020)
- [A Context-Integrated Transformer-Based Neural Network for Auction Design](https://mlsanity.com/paper/2201.12489/) (2022)
- [Reinforcement Mechanism Design, with Applications to Dynamic Pricing in Sponsored Search Auctions](https://mlsanity.com/paper/1711.10279/) (2017)
- [Combinatorial Auctions via Posted Prices](https://mlsanity.com/paper/1411.4916/) (2014)
- [Simple Mechanisms for Subadditive Buyers via Duality](https://mlsanity.com/paper/1611.06910/) (2016)
- [Reinforcement Learning of Sequential Price Mechanisms](https://mlsanity.com/paper/2010.01180/) (2020)
- [Computing Optimal Equilibria and Mechanisms via Learning in Zero-Sum Extensive-Form Games](https://mlsanity.com/paper/2306.05216/) (2023)
- [Prophet Inequalities Made Easy: Stochastic Optimization by Pricing Non-Stochastic Inputs](https://mlsanity.com/paper/1612.03161/) (2016)
- [On the Complexity of Dynamic Mechanism Design](https://mlsanity.com/paper/1407.5373/) (2014)
- [Sequential Mechanisms with ex-post Participation Guarantees](https://mlsanity.com/paper/1603.07229/) (2016)
- [Improved Truthful Mechanisms for Combinatorial Auctions with Submodular Bidders](https://mlsanity.com/paper/1911.02716/) (2019)
- [Computing Simple Mechanisms: Lift-and-Round over Marginal Reduced Forms](https://mlsanity.com/paper/2111.03962/) (2021)
- [Online Combinatorial Auctions for Resource Allocation with Supply Costs and Capacity Limits](https://mlsanity.com/paper/2209.07035/) (2022)

## Built on by (most cited first, 1 of 1)

- [MIRT: Transformers for Truthful Generative Auctions with Whole-feed Permutation Externalities](https://mlsanity.com/paper/2610.06559/) (2026)

## Abstract (arXiv)

Revenue-optimal auction design is a challenging problem with significant theoretical and practical implications. Sequential auction mechanisms, known for their simplicity and strong strategyproofness guarantees, are often limited by theoretical results that are largely existential, except for certain restrictive settings. Although traditional reinforcement learning methods such as Proximal Policy Optimization (PPO) and Soft Actor-Critic (SAC) are applicable in this domain, they struggle with computational demands and convergence issues when dealing with large and continuous action spaces. In light of this and recognizing that we can model transitions differentiable for our settings, we propose using a new reinforcement learning framework tailored for sequential combinatorial auctions that leverages first-order gradients. Our extensive evaluations show that our approach achieves significant improvement in revenue over both analytical baselines and standard reinforcement learning algorithms. Furthermore, we scale our approach to scenarios involving up to 50 agents and 50 items, demonstrating its applicability in complex, real-world auction settings. As such, this work advances the computational tools available for auction design and contributes to bridging the gap between theoretical results and practical implementations in sequential auction design.