# Q-Regularized Generative Auto-Bidding: From Suboptimal Trajectories to Optimal Policies

arXiv: https://arxiv.org/abs/2601.02754 · HTML: https://mlsanity.com/paper/2601.02754/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in RL theory and sample complexity, one of 15 directions in Deep reinforcement learning methods.
- Builds on 30 of its 52 references that are in the map.
- First posted on arXiv 6 Jan 2026.

## Placed in

- Deep reinforcement learning methods / [RL theory and sample complexity](https://mlsanity.com/deeprl/rl-theory-and-sample-complexity/)

## Builds on (most cited first, 30 of 30)

- [Continuous control with deep reinforcement learning](https://mlsanity.com/paper/1509.02971/) (2015)
- [Trust Region Policy Optimization](https://mlsanity.com/paper/1502.05477/) (2015)
- [Addressing Function Approximation Error in Actor-Critic Methods](https://mlsanity.com/paper/1802.09477/) (2018)
- [Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems](https://mlsanity.com/paper/2005.01643/) (2020)
- [Conservative Q-Learning for Offline Reinforcement Learning](https://mlsanity.com/paper/2006.04779/) (2020)
- [Deep Reinforcement Learning with Double Q-learning](https://mlsanity.com/paper/1509.06461/) (2015)
- [Decision Transformer: Reinforcement Learning via Sequence Modeling](https://mlsanity.com/paper/2106.01345/) (2021)
- [Offline Reinforcement Learning with Implicit Q-Learning](https://mlsanity.com/paper/2110.06169/) (2021)
- [Off-Policy Deep Reinforcement Learning without Exploration](https://mlsanity.com/paper/1812.02900/) (2018)
- [A Minimalist Approach to Offline Reinforcement Learning](https://mlsanity.com/paper/2106.06860/) (2021)
- [Monte Carlo Tree Search: A Review of Recent Modifications and Applications](https://mlsanity.com/paper/2103.04931/) (2021)
- [Online Convex Optimization with Stochastic Constraints](https://mlsanity.com/paper/1708.03741/) (2017)
- [Real-Time Bidding by Reinforcement Learning in Display Advertising](https://mlsanity.com/paper/1701.02490/) (2017)
- [Budget Constrained Bidding by Model-free Reinforcement Learning in Display Advertising](https://mlsanity.com/paper/1802.08365/) (2018)
- [Mean Absolute Percentage Error for regression models](https://mlsanity.com/paper/1605.02541/) (2016)
- [Bid Optimization by Multivariable Control in Display Advertising](https://mlsanity.com/paper/1905.10928/) (2019)
- [DiffStitch: Boosting Offline Reinforcement Learning with Diffusion-based Trajectory Stitching](https://mlsanity.com/paper/2402.02439/) (2024)
- [Deep Reinforcement Learning for Sponsored Search Real-time Bidding](https://mlsanity.com/paper/1803.00259/) (2018)
- [Multi-Task Deep Recommender Systems: A Survey](https://mlsanity.com/paper/2302.03525/) (2023)
- [Sustainable Online Reinforcement Learning for Auto-bidding](https://mlsanity.com/paper/2210.07006/) (2022)
- [AuctionNet: A Novel Benchmark for Decision-Making in Large-Scale Games](https://mlsanity.com/paper/2412.10798/) (2024)
- [Auto-bidding and Auctions in Online Advertising: A Survey](https://mlsanity.com/paper/2408.07685/) (2024)
- [Generative Auto-Bidding with Value-Guided Explorations](https://mlsanity.com/paper/2504.14587/) (2025)
- [Feedback Control of Real-Time Display Advertising](https://mlsanity.com/paper/1603.01055/) (2016)
- [A Cooperative-Competitive Multi-Agent Framework for Auto-bidding in Online Advertising](https://mlsanity.com/paper/2106.06224/) (2021)
- [RTBAgent: A LLM-based Agent System for Real-Time Bidding](https://mlsanity.com/paper/2502.00792/) (2025)
- [Trajectory-wise Iterative Reinforcement Learning Framework for Auto-bidding](https://mlsanity.com/paper/2402.15102/) (2024)
- [Offline Reinforcement Learning for Optimizing Production Bidding Policies](https://mlsanity.com/paper/2310.09426/) (2023)
- [GAS: Generative Auto-bidding with Post-training Search](https://mlsanity.com/paper/2412.17018/) (2024)
- [Optimal Return-to-Go Guided Decision Transformer for Auto-Bidding in Advertisement](https://mlsanity.com/paper/2506.21956/) (2025)

## Abstract (arXiv)

With the rapid development of e-commerce, auto-bidding has become a key asset in optimizing advertising performance under diverse advertiser environments. The current approaches focus on reinforcement learning (RL) and generative models. These efforts imitate offline historical behaviors by utilizing a complex structure with expensive hyperparameter tuning. The suboptimal trajectories further exacerbate the difficulty of policy learning. To address these challenges, we proposes QGA, a novel Q-value regularized Generative Auto-bidding method. In QGA, we propose to plug a Q-value regularization with double Q-learning strategy into the Decision Transformer backbone. This design enables joint optimization of policy imitation and action-value maximization, allowing the learned bidding policy to both leverage experience from the dataset and alleviate the adverse impact of the suboptimal trajectories. Furthermore, to safely explore the policy space beyond the data distribution, we propose a Q-value guided dual-exploration mechanism, in which the DT model is conditioned on multiple return-to-go targets and locally perturbed actions. This entire exploration process is dynamically guided by the aforementioned Q-value module, which provides principled evaluation for each candidate action. Experiments on public benchmarks and simulation environments demonstrate that QGA consistently achieves superior or highly competitive results compared to existing alternatives. Notably, in large-scale real-world A/B testing, QGA achieves a 3.27% increase in Ad GMV and a 2.49% improvement in Ad ROI.