mlsanityData to 14 Oct 2026

FAST-Q: Fast-track Exploration with Adversarially Balanced State Representations for Counterfactual Action Estimation in Offline Reinforcement Learning

arXiv 2504.21383 ↗First posted 30 Apr 2025cs.LG cs.AIcited by 0

On the map: Offline RL and imitation learning from data → Offline RL algorithms

Abstract (arXiv)

Recent advancements in state-of-the-art (SOTA) offline reinforcement learning (RL) have primarily focused on addressing function approximation errors, which contribute to the overestimation of Q-values for out-of-distribution actions, a challenge that static datasets exacerbate. However, high stakes applications such as recommendation systems in online gaming, introduce further complexities due to player's psychology (intent) driven by gameplay experiences and the inherent volatility on the platform. These factors create highly sparse, partially overlapping state spaces across policies, further influenced by the experiment path selection logic which biases state spaces towards specific policies. Current SOTA methods constrain learning from such offline data by clipping known counterfactual actions as out-of-distribution due to poor generalization across unobserved states. Further aggravating conservative Q-learning and necessitating more online exploration. FAST-Q introduces a novel approach that (1) leverages Gradient Reversal Learning to construct balanced state representations, regularizing the policy-specific bias between the player's state and action thereby enabling counterfactual estimation; (2) supports offline counterfactual exploration in parallel with static data exploitation; and (3) proposes a Q-value decomposition strategy for multi-objective optimization, facilitating explainable recommendations over short and long-term objectives. These innovations demonstrate superiority of FAST-Q over prior SOTA approaches and demonstrates at least 0.15 percent increase in player returns, 2 percent improvement in lifetime value (LTV), 0.4 percent enhancement in the recommendation driven engagement, 2 percent improvement in the player's platform dwell time and an impressive 10 percent reduction in the costs associated with the recommendation, on our volatile gaming platform.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Offline RL algorithms, one of 13 directions in Offline RL and imitation learning from data. Builds on 16 of its 38 references that are in the map. First posted on arXiv 30 Apr 2025.

Builds on

16 of 38 references.

Closest ancestors · 4

Same direction, newest first

  1. 2021DR3: Value-Based Deep Reinforcement Learning Requires Explicit Regularization9 Dec
  2. 2021Offline Reinforcement Learning with Implicit Q-Learning12 Oct
  3. 2020Conservative Q-Learning for Offline Reinforcement Learning8 Jun
  4. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learning15 Apr

Other references · 12

  1. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  2. 2015Domain-Adversarial Training of Neural Networkscited by 3,044
  3. 2018UMAP: Uniform Manifold Approximation and Projection for Dimension Reductioncited by 2,898
  4. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,417
  5. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,241
  6. 2019Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reductioncited by 840
  7. 2017Neural Attentive Session-based Recommendationcited by 237
  8. 2019A Generalized Algorithm for Multi-Objective Reinforcement Learning and Policy Adaptationcited by 177
  9. 2015A Theoretically Grounded Application of Dropout in Recurrent Neural Networkscited by 161
  10. 2020Estimating Counterfactual Treatment Outcomes over Time Through Adversarially Balanced Representationscited by 155
  11. 2019Reinforcement Learning to Optimize Long-term User Engagement in Recommender Systemscited by 90
  12. 2019Multi-Domain Adversarial Learningcited by 19

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2023Offline Reinforcement Learning with Imbalanced Datasets
  2. 2023A Perspective of Q-value Estimation on Offline-to-Online Reinforcement Learning
  3. 2024State-Constrained Offline Reinforcement Learning
  4. 2022Mildly Conservative Q-Learning for Offline Reinforcement Learning
  5. 2022Offline Q-Learning on Diverse Multi-Task Data Both Scales And Generalizes
  6. 2023Offline Reinforcement Learning with On-Policy Q-Function Regularization
  7. 2021Offline Reinforcement Learning with Reverse Model-based Imagination
  8. 2024Strategically Conservative Q-Learning