# Generative Auto-Bidding with Value-Guided Explorations

arXiv: https://arxiv.org/abs/2504.14587 · HTML: https://mlsanity.com/paper/2504.14587/

## Key facts

- Cited by 23 papers in the map (data through 14 Oct 2026).
- Builds on 31 of its 58 references that are in the map.
- First posted on arXiv 20 Apr 2025.

## Builds on (most cited first, 30 of 31)

- [Playing Atari with Deep Reinforcement Learning](https://mlsanity.com/paper/1312.5602/) (2013)
- [Decision Transformer: Reinforcement Learning via Sequence Modeling](https://mlsanity.com/paper/2106.01345/) (2021)
- [Offline Reinforcement Learning with Implicit Q-Learning](https://mlsanity.com/paper/2110.06169/) (2021)
- [Off-Policy Deep Reinforcement Learning without Exploration](https://mlsanity.com/paper/1812.02900/) (2018)
- [Offline Reinforcement Learning with Fisher Divergence Critic Regularization](https://mlsanity.com/paper/2103.08050/) (2021)
- [Online Decision Transformer](https://mlsanity.com/paper/2202.05607/) (2022)
- [Bridging the Gap Between Value and Policy Based Reinforcement Learning](https://mlsanity.com/paper/1702.08892/) (2017)
- [Offline RL with No OOD Actions: In-Sample Learning via Implicit Value Regularization](https://mlsanity.com/paper/2303.15810/) (2023)
- [Online Convex Optimization with Stochastic Constraints](https://mlsanity.com/paper/1708.03741/) (2017)
- [Real-Time Bidding by Reinforcement Learning in Display Advertising](https://mlsanity.com/paper/1701.02490/) (2017)
- [Recommendations with Negative Feedback via Pairwise Deep Reinforcement Learning](https://mlsanity.com/paper/1802.06501/) (2018)
- [Constrained Decision Transformer for Offline Safe Reinforcement Learning](https://mlsanity.com/paper/2302.07351/) (2023)
- [Budget Constrained Bidding by Model-free Reinforcement Learning in Display Advertising](https://mlsanity.com/paper/1802.08365/) (2018)
- [DEAR: Deep Reinforcement Learning for Online Advertising Impression in Recommender Systems](https://mlsanity.com/paper/1909.03602/) (2019)
- [LLM4Rerank: LLM-based Auto-Reranking Framework for Recommendations](https://mlsanity.com/paper/2406.12433/) (2024)
- [Deep Reinforcement Learning for Sponsored Search Real-time Bidding](https://mlsanity.com/paper/1803.00259/) (2018)
- [Real-time Bidding for Online Advertising: Measurement and Analysis](https://mlsanity.com/paper/1306.6542/) (2013)
- [Multi-Task Deep Recommender Systems: A Survey](https://mlsanity.com/paper/2302.03525/) (2023)
- [Sustainable Online Reinforcement Learning for Auto-bidding](https://mlsanity.com/paper/2210.07006/) (2022)
- [AuctionNet: A Novel Benchmark for Decision-Making in Large-Scale Games](https://mlsanity.com/paper/2412.10798/) (2024)
- [Jointly Learning to Recommend and Advertise](https://mlsanity.com/paper/2003.00097/) (2020)
- [Multi-Task Recommendations with Reinforcement Learning](https://mlsanity.com/paper/2302.03328/) (2023)
- [Auto-bidding and Auctions in Online Advertising: A Survey](https://mlsanity.com/paper/2408.07685/) (2024)
- [Deep reinforcement learning for search, recommendation, and online advertising: a survey](https://mlsanity.com/paper/1812.07127/) (2018)
- [AutoDenoise: Automatic Data Instance Denoising for Recommendations](https://mlsanity.com/paper/2303.06611/) (2023)
- [A Cooperative-Competitive Multi-Agent Framework for Auto-bidding in Online Advertising](https://mlsanity.com/paper/2106.06224/) (2021)
- [HAMUR: Hyper Adapter for Multi-Domain Recommendation](https://mlsanity.com/paper/2309.06217/) (2023)
- [Exploration and Regularization of the Latent Action Space in Recommendation](https://mlsanity.com/paper/2302.03431/) (2023)
- [Accelerating Offline Reinforcement Learning Application in Real-Time Bidding and Recommendation: Potential Use of Simulation](https://mlsanity.com/paper/2109.08331/) (2021)
- [Sequential Recommendation for Optimizing Both Immediate Feedback and Long-term Retention](https://mlsanity.com/paper/2404.03637/) (2024)

## Built on by (most cited first, 23 of 23)

- [NEZHA: A Zero-sacrifice and Hyperspeed Decoding Architecture for Generative Recommendations](https://mlsanity.com/paper/2511.18793/) (2025)
- [Generative Auto-Bidding in Large-Scale Competitive Auctions via Diffusion Completer-Aligner](https://mlsanity.com/paper/2509.03348/) (2025)
- [GFlowGR: Fine-tuning Generative Recommendation Frameworks with Generative Flow Networks](https://mlsanity.com/paper/2506.16114/) (2025)
- [Generative Large-Scale Pre-trained Models for Automated Ad Bidding Optimization](https://mlsanity.com/paper/2508.02002/) (2025)
- [Multi-task Offline Reinforcement Learning for Online Advertising in Recommender Systems](https://mlsanity.com/paper/2506.23090/) (2025)
- [Expert-Guided Diffusion Planner for Auto-Bidding](https://mlsanity.com/paper/2508.08687/) (2025)
- [Autobidding Arena: unified evaluation of the classical and RL-based autobidding algorithms](https://mlsanity.com/paper/2510.19357/) (2025)
- [C2:Cross learning module enhanced decision transformer with Constraint-aware loss for auto-bidding](https://mlsanity.com/paper/2601.20257/) (2026)
- [PRO-Bid: Pareto-Prioritized Regret Optimization for Constraint-Aware Generative Auto-Bidding](https://mlsanity.com/paper/2602.08261/) (2026)
- [LBM: Hierarchical Large Auto-Bidding Model via Reasoning and Acting](https://mlsanity.com/paper/2603.05134/) (2026)
- [PlatformBid: An Auto-Bidding Benchmark from a Unified Advertising Platform's Perspective](https://mlsanity.com/paper/2607.27265/) (2026)
- [HALO: Hindsight-Augmented Learning for Online Auto-Bidding](https://mlsanity.com/paper/2508.03267/) (2025)
- [Generative Sequential Notification Optimization via Multi-Objective Decision Transformers](https://mlsanity.com/paper/2509.02458/) (2025)
- [Q-Regularized Generative Auto-Bidding: From Suboptimal Trajectories to Optimal Policies](https://mlsanity.com/paper/2601.02754/) (2026)
- [AHBid: An Adaptable Hierarchical Bidding Framework for Cross-Channel Advertising](https://mlsanity.com/paper/2602.22650/) (2026)
- [Knowledge-informed Bidding with Dual-process Control for Online Advertising](https://mlsanity.com/paper/2603.04920/) (2026)
- [JD-BP: A Joint-Decision Generative Framework for Auto-Bidding and Pricing](https://mlsanity.com/paper/2604.05845/) (2026)
- [Generative Auto-Bidding with Unified Modeling and Exploration](https://mlsanity.com/paper/2605.19457/) (2026)
- [Constrained Auto-Bidding via Generative Response Modeling](https://mlsanity.com/paper/2605.27811/) (2026)
- [DRIVE: Distributional and Retrieval-Augmented Bidding with Value Evaluation](https://mlsanity.com/paper/2606.14192/) (2026)
- [AIGB-R1: Self-Evolving Generative Auto-Bidding via Hierarchical Planner-Executor Optimization](https://mlsanity.com/paper/2607.17281/) (2026)
- [Beyond Single-Episode Optimization: Sliding-Window Aware Generative Auto-Bidding for Long-Term Advertising Effectiveness](https://mlsanity.com/paper/2607.25233/) (2026)
- [A Self-Triggered Agentic Push Recommendation System](https://mlsanity.com/paper/2608.01949/) (2026)

## Abstract (arXiv)

Auto-bidding, with its strong capability to optimize bidding decisions within dynamic and competitive online environments, has become a pivotal strategy for advertising platforms. Existing approaches typically employ rule-based strategies or Reinforcement Learning (RL) techniques. However, rule-based strategies lack the flexibility to adapt to time-varying market conditions, and RL-based methods struggle to capture essential historical dependencies and observations within Markov Decision Process (MDP) frameworks. Furthermore, these approaches often face challenges in ensuring strategy adaptability across diverse advertising objectives. Additionally, as offline training methods are increasingly adopted to facilitate the deployment and maintenance of stable online strategies, the issues of documented behavioral patterns and behavioral collapse resulting from training on fixed offline datasets become increasingly significant. To address these limitations, this paper introduces a novel offline Generative Auto-bidding framework with Value-Guided Explorations (GAVE). GAVE accommodates various advertising objectives through a score-based Return-To-Go (RTG) module. Moreover, GAVE integrates an action exploration mechanism with an RTG-based evaluation method to explore novel actions while ensuring stability-preserving updates. A learnable value function is also designed to guide the direction of action exploration and mitigate Out-of-Distribution (OOD) problems. Experimental results on two offline datasets and real-world deployments demonstrate that GAVE outperforms state-of-the-art baselines in both offline evaluations and online A/B tests. By applying the core methods of this framework, we proudly secured first place in the NeurIPS 2024 competition, 'AIGB Track: Learning Auto-Bidding Agents with Generative Models'.