mlsanityData to 7 Oct 2026

State Advantage Weighting for Offline RL

arXiv 2210.04251 ↗First posted 9 Oct 2022cs.LG cs.AIcited by 5

On the map: Offline RL and imitation learning from data → Offline RL algorithms

Abstract (arXiv)

We present state advantage weighting for offline reinforcement learning (RL). In contrast to action advantage A(s,a) that we commonly adopt in QSA learning, we leverage state advantage A(s,s^prime) and QSS learning for offline RL, hence decoupling the action from values. We expect the agent can get to the high-reward state and the action is determined by how the agent can get to that corresponding state. Experiments on D4RL datasets show that our proposed method can achieve remarkable performance against the common baselines. Furthermore, our method shows good generalization capability when transferring from offline to online.

Key facts. Cited by 5 papers in the map (data through 7 Oct 2026). Placed in Offline RL algorithms, one of 13 directions in Offline RL and imitation learning from data. Builds on 45 of its 65 references that are in the map. First posted on arXiv 9 Oct 2022.

Builds on

45 of 65 references.

Closest ancestors · 20

Same direction, newest first

  1. 2022Mildly Conservative Q-Learning for Offline Reinforcement Learning9 Jun
  2. 2022Learning Value Functions from Undirected State-only Experience26 Apr
  3. 2022Pessimistic Bootstrapping for Uncertainty-Driven Offline Reinforcement Learning23 Feb
  4. 2021Offline Reinforcement Learning with Value-based Episodic Memory19 Oct
  5. 2021Offline Reinforcement Learning with Implicit Q-Learning12 Oct
  6. 2021Uncertainty-Based Offline Reinforcement Learning with Diversified Q-Ensemble4 Oct
  7. 2021Offline RL Without Off-Policy Evaluation16 Jun
  8. 2021A Minimalist Approach to Offline Reinforcement Learning12 Jun
  9. 2021Uncertainty Weighted Actor-Critic for Offline Reinforcement Learning17 May
  10. 2021Offline Reinforcement Learning with Fisher Divergence Critic Regularization14 Mar
  11. 2021Continuous Doubly Constrained Batch Reinforcement Learning18 Feb
  12. 2021COMBO: Conservative Offline Model-Based Policy Optimization16 Feb
8 more
  1. 2020PLAS: Latent Action Space for Offline Reinforcement Learning14 Nov
  2. 2020EMaQ: Expected-Max Q-Learning Operator for Simple Yet Effective Offline and Online RL21 Jul
  3. 2020Conservative Q-Learning for Offline Reinforcement Learning8 Jun
  4. 2020MOPO: Model-based Offline Policy Optimization27 May
  5. 2020MOReL : Model-Based Offline Reinforcement Learning12 May
  6. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems4 May
  7. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learning15 Apr
  8. 2019Behavior Regularized Offline Reinforcement Learning26 Nov

Other references · 25

  1. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  2. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,302
  3. 2016OpenAI Gymcited by 1,684
  4. 2021Decision Transformer: Reinforcement Learning via Sequence Modelingcited by 1,528
  5. 2019Can You Trust Your Model's Uncertainty? Evaluating Predictive Uncertainty Under Dataset Shiftcited by 1,274
  6. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,227
  7. 2019Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reductioncited by 833
  8. 2015Dueling Network Architectures for Deep Reinforcement Learningcited by 769
  9. 2019Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learningcited by 750
  10. 2017A Distributional Perspective on Reinforcement Learningcited by 743
  11. 2020AWAC: Accelerating Online Reinforcement Learning with Offline Datasetscited by 647
  12. 2021Offline Reinforcement Learning as One Big Sequence Modeling Problemcited by 636
13 more
  1. 2022Diffusion Policies as an Expressive Policy Class for Offline Reinforcement Learningcited by 543
  2. 2019DualDICE: Behavior-Agnostic Estimation of Discounted Stationary Distribution Correctionscited by 254
  3. 2015An Emphatic Approach to the Problem of Off-policy Temporal-Difference Learningcited by 103
  4. 2019Recent Advances in Imitation Learning from Observationcited by 92
  5. 2019BAIL: Best-Action Imitation Learning for Batch Deep Reinforcement Learningcited by 90
  6. 2019Provably Efficient Imitation Learning from Observation Alonecited by 78
  7. 2021Believe What You See: Implicit Constraint Approach for Offline Multi-Agent Reinforcement Learningcited by 78
  8. 2019Off-Policy Deep Reinforcement Learning by Bootstrapping the Covariate Shiftcited by 47
  9. 2021UMBRELLA: Uncertainty-Aware Model-Based Offline Reinforcement Learning Leveraging Planningcited by 23
  10. 2020Estimating Q(s,s') with Deep Deterministic Dynamics Gradientscited by 12
  11. 2022IL-flOw: Imitation Learning from Observation using Normalizing Flowscited by 12
  12. 2019Sample-efficient Adversarial Imitation Learning from Observationcited by 6
  13. 2020State Action Separable Reinforcement Learningcited by 2

Built on by

Cited by year

Citing papers per year: 2024: 5.

5 papers.

  1. 2024Exploration and Anti-Exploration with Distributional Random Network Distillationcited by 32
  2. 2024ODRL: A Benchmark for Off-Dynamics Reinforcement Learningcited by 25
  3. 2024Cross-Domain Policy Adaptation by Capturing Representation Mismatchcited by 24
  4. 2024SEABO: A Simple Search-Based Method for Offline Imitation Learningcited by 14
  5. 2024CDSA: Conservative Denoising Score-based Algorithm for Offline Reinforcement Learningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023Goal-conditioned Offline Reinforcement Learning through State Space Partitioning
  2. 2023Cal-QL: Calibrated Offline RL Pre-Training for Efficient Online Fine-Tuning
  3. 2025A Tutorial: An Intuitive Explanation of Offline Reinforcement Learning Theory
  4. 2024SAMG: Offline-to-Online Reinforcement Learning via State-Action-Conditional Offline Model Guidance
  5. 2024State-Constrained Offline Reinforcement Learning
  6. 2023Exploiting Generalization in Offline Reinforcement Learning via Unseen State Augmentations
  7. 2023Offline Reinforcement Learning with Imbalanced Datasets
  8. 2020Hyperparameter Selection for Offline Reinforcement Learning