mlsanityData to 14 Oct 2026

Permutation Equivariant Model-based Offline Reinforcement Learning for Auto-bidding

arXiv 2506.17919 ↗First posted 22 Jun 2025cs.LG cs.AIcited by 1

On the map: Offline RL and imitation learning from data → Offline RL algorithms

Abstract (arXiv)

Reinforcement learning (RL) for auto-bidding has shifted from using simplistic offline simulators (Simulation-based RL Bidding, SRLB) to offline RL on fixed real datasets (Offline RL Bidding, ORLB). However, ORLB policies are limited by the dataset's state space coverage, offering modest gains. While SRLB expands state coverage, its simulator-reality gap risks misleading policies. This paper introduces Model-based RL Bidding (MRLB), which learns an environment model from real data to bridge this gap. MRLB trains policies using both real and model-generated data, expanding state coverage beyond ORLB. To ensure model reliability, we propose: 1) A permutation equivariant model architecture for better generalization, and 2) A robust offline Q-learning method that pessimistically penalizes model errors. These form the Permutation Equivariant Model-based Offline RL (PE-MORL) algorithm. Real-world experiments show that PE-MORL outperforms state-of-the-art auto-bidding methods.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Placed in Offline RL algorithms, one of 13 directions in Offline RL and imitation learning from data. Builds on 16 of its 29 references that are in the map. First posted on arXiv 22 Jun 2025.

Builds on

16 of 29 references.

Closest ancestors · 1

Same direction, newest first

  1. 2020MOPO: Model-based Offline Policy Optimization27 May

Other references · 15

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
  3. 2017What Uncertainties Do We Need in Bayesian Deep Learning for Computer Vision?cited by 2,352
  4. 2017Deep Setscited by 1,421
  5. 2020AWAC: Accelerating Online Reinforcement Learning with Offline Datasetscited by 671
  6. 2019Model-Based Reinforcement Learning for Ataricited by 502
  7. 2019Monte Carlo Gradient Estimation in Machine Learningcited by 283
  8. 2018Algorithmic Framework for Model-based Deep Reinforcement Learning with Theoretical Guaranteescited by 140
  9. 2017Real-Time Bidding by Reinforcement Learning in Display Advertisingcited by 91
  10. 2018Real-Time Bidding with Multi-Agent Reinforcement Learning in Display Advertisingcited by 65
  11. 2018Budget Constrained Bidding by Model-free Reinforcement Learning in Display Advertisingcited by 61
  12. 2022Sustainable Online Reinforcement Learning for Auto-biddingcited by 29
3 more
  1. 2022A Context-Integrated Transformer-Based Neural Network for Auction Designcited by 19
  2. 2021Permutation Invariant Policy Optimization for Mean-Field Multi-Agent Reinforcement Learning: A Principled Approachcited by 14
  3. 2023Offline Reinforcement Learning for Optimizing Production Bidding Policiescited by 8

Built on by

Cited by year

Citing papers per year: 2025: 1.

1 paper.

  1. 2025Enhancing Generative Auto-bidding with Offline Reward Evaluation and Policy Searchcited by 1

Similar, not cited

Close in wording, no citation either way

  1. 2023Equivariant Data Augmentation for Generalization in Offline Reinforcement Learning
  2. 2024Trajectory-wise Iterative Reinforcement Learning Framework for Auto-bidding
  3. 2020MOReL : Model-Based Offline Reinforcement Learning
  4. 2022Oracle Inequalities for Model Selection in Offline Reinforcement Learning
  5. 2023A Survey on Offline Model-Based Reinforcement Learning
  6. 2024Equivariant Offline Reinforcement Learning
  7. 2023Offline Reinforcement Learning with Imbalanced Datasets
  8. 2025Enhancing Offline Model-Based RL via Active Model Selection: A Bayesian Optimization Perspective