# Learning to Allocate Incentives for Incentivized Advertising via Offline Model-Based Reinforcement Learning

arXiv: https://arxiv.org/abs/2608.28065 · HTML: https://mlsanity.com/paper/2608.28065/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Decision transformers and sequence modeling, one of 13 directions in Offline RL and imitation learning from data.
- Builds on 24 of its 34 references that are in the map.
- First posted on arXiv 28 Aug 2026.

## Placed in

- Offline RL and imitation learning from data / [Decision transformers and sequence modeling](https://mlsanity.com/offlinerl/decision-transformers-and-sequence-modeling/)

## Builds on (most cited first, 24 of 24)

- [Conservative Q-Learning for Offline Reinforcement Learning](https://mlsanity.com/paper/2006.04779/) (2020)
- [Decision Transformer: Reinforcement Learning via Sequence Modeling](https://mlsanity.com/paper/2106.01345/) (2021)
- [D4RL: Datasets for Deep Data-Driven Reinforcement Learning](https://mlsanity.com/paper/2004.07219/) (2020)
- [Offline Reinforcement Learning with Implicit Q-Learning](https://mlsanity.com/paper/2110.06169/) (2021)
- [A Minimalist Approach to Offline Reinforcement Learning](https://mlsanity.com/paper/2106.06860/) (2021)
- [MOPO: Model-based Offline Policy Optimization](https://mlsanity.com/paper/2005.13239/) (2020)
- [Estimation and Inference of Heterogeneous Treatment Effects using Random Forests](https://mlsanity.com/paper/1510.04342/) (2015)
- [Introduction to Multi-Armed Bandits](https://mlsanity.com/paper/1904.07272/) (2019)
- [COMBO: Conservative Offline Model-Based Policy Optimization](https://mlsanity.com/paper/2102.08363/) (2021)
- [Batch Policy Learning under Constraints](https://mlsanity.com/paper/1903.08738/) (2019)
- [Top-K Off-Policy Correction for a REINFORCE Recommender System](https://mlsanity.com/paper/1812.02353/) (2018)
- [Deep Reinforcement Learning for Page-wise Recommendations](https://mlsanity.com/paper/1805.02343/) (2018)
- [Real-Time Bidding by Reinforcement Learning in Display Advertising](https://mlsanity.com/paper/1701.02490/) (2017)
- [Is Value Learning Really the Main Bottleneck in Offline RL?](https://mlsanity.com/paper/2406.09329/) (2024)
- [Budget Constrained Bidding by Model-free Reinforcement Learning in Display Advertising](https://mlsanity.com/paper/1802.08365/) (2018)
- [OneTrans: Unified Feature Interaction and Sequence Modeling with One Transformer in Industrial Recommender](https://mlsanity.com/paper/2510.26104/) (2025)
- [A Unified Framework for Marketing Budget Allocation](https://mlsanity.com/paper/1902.01128/) (2019)
- [Model-based Offline Reinforcement Learning with Lower Expectile Q-Learning](https://mlsanity.com/paper/2407.00699/) (2024)
- [Spending Money Wisely: Online Electronic Coupon Allocation based on Real-Time User Intent Detection](https://mlsanity.com/paper/2008.09982/) (2020)
- [Cost-Effective Incentive Allocation via Structured Counterfactual Inference](https://mlsanity.com/paper/1902.02495/) (2019)
- [End-to-End Cost-Effective Incentive Recommendation under Budget Constraint with Uplift Modeling](https://mlsanity.com/paper/2408.11623/) (2024)
- [A Framework for Multi-stage Bonus Allocation in meal delivery Platform](https://mlsanity.com/paper/2202.10695/) (2022)
- [Revenue Maximization in Incentivized Social Advertising](https://mlsanity.com/paper/1612.00531/) (2016)
- [BCRLSP: An Offline Reinforcement Learning Framework for Sequential Targeted Promotion](https://mlsanity.com/paper/2207.07790/) (2022)

## Abstract (arXiv)

Complete your ad view and grab a 5-cent bonus! In incentivized advertising, a platform promises users a bonus before observing downstream ad revenue, encouraging them to click and complete ads. It must balance the incentive promised in advance against the revenue realized afterward: insufficient incentives forfeit monetization opportunities, whereas excessive incentives reduce net profit. Because current incentives may also shape user expectations and future engagement, incentive allocation is a sequential decision problem with delayed revenue, cost sensitivity, and carryover effects. Existing work has not studied decision-making algorithms for this setting. Auto-bidding assumes available ad opportunities, while targeted promotion optimizes incentives outside the ad monetization pipeline. We formulate the problem as an MDP and develop an offline model-based RL framework for cost-controllable sequential incentive allocation. It learns a world model of user feedback and ad revenue, then performs conservative policy optimization. An independent counterfactual scorer evaluates each learned policy on held-out logs, enabling pre-launch selection without costly online exposure. Experiments on large-scale industrial data and online A/B tests show that the scorer provides a stable offline signal. The deployment path from causal inference to offline RL and then Offline-MBRL further validates the framework: MB-IQL improves per-user net profit by 7.96% over TD3+BC, whereas reverting to plain IQL reduces it by 6.56% (both (p<0.0001)).