mlsanityData to 7 Oct 2026

A2PO: Towards Effective Offline Reinforcement Learning from an Advantage-aware Perspective

arXiv 2403.07262 ↗First posted 12 Mar 2024cs.LG cs.AIcited by 10

On the map: Offline RL and imitation learning from data → Offline RL algorithms

Abstract (arXiv)

Offline reinforcement learning endeavors to leverage offline datasets to craft effective agent policy without online interaction, which imposes proper conservative constraints with the support of behavior policies to tackle the out-of-distribution problem. However, existing works often suffer from the constraint conflict issue when offline datasets are collected from multiple behavior policies, i.e., different behavior policies may exhibit inconsistent actions with distinct returns across the state space. To remedy this issue, recent advantage-weighted methods prioritize samples with high advantage values for agent training while inevitably ignoring the diversity of behavior policy. In this paper, we introduce a novel Advantage-Aware Policy Optimization (A2PO) method to explicitly construct advantage-aware policy constraints for offline learning under mixed-quality datasets. Specifically, A2PO employs a conditional variational auto-encoder to disentangle the action distributions of intertwined behavior policies by modeling the advantage values of all training data as conditional variables. Then the agent can follow such disentangled action distribution constraints to optimize the advantage-aware policy towards high advantage values. Extensive experiments conducted on both the single-quality and mixed-quality datasets of the D4RL benchmark demonstrate that A2PO yields results superior to the counterparts. Our code is available at https://github.com/Plankson/A2PO

Key facts. Cited by 10 papers in the map (data through 7 Oct 2026). Placed in Offline RL algorithms, one of 13 directions in Offline RL and imitation learning from data. Builds on 47 of its 61 references that are in the map. First posted on arXiv 12 Mar 2024.

Builds on

47 of 61 references.

Closest ancestors · 25

Same direction, newest first

  1. 2023Train Once, Get a Family: State-Adaptive Balances for Offline-to-Online Reinforcement Learning27 Oct
  2. 2023Beyond Uniform Sampling: Offline Reinforcement Learning with Imbalanced Datasets6 Oct
  3. 2023Model-based Offline Policy Optimization with Adversarial Network5 Sep
  4. 2023Harnessing Mixed Offline Reinforcement Learning Datasets via Trajectory Weighting22 Jun
  5. 2023Policy Regularization with Dataset Constraint for Offline Reinforcement Learning11 Jun
  6. 2023Offline RL with No OOD Actions: In-Sample Learning via Implicit Value Regularization28 Mar
  7. 2023On the Sample Complexity of Vanilla Model-Based Offline Reinforcement Learning with Dependent Samples7 Mar
  8. 2023The In-Sample Softmax for Offline Reinforcement Learning28 Feb
  9. 2023Behavior Proximal Policy Optimization22 Feb
  10. 2022Confidence-Conditioned Value Functions for Offline Reinforcement Learning8 Dec
  11. 2022Leveraging Offline Data in Online Reinforcement Learning9 Nov
  12. 2022Offline RL With Realistic Datasets: Heteroskedasticity and Support Constraints2 Nov
13 more
  1. 2022Boosting Offline Reinforcement Learning via Data Rebalancing17 Oct
  2. 2022Semi-Supervised Offline Reinforcement Learning with Action-Free Trajectories12 Oct
  3. 2022Mildly Conservative Q-Learning for Offline Reinforcement Learning9 Jun
  4. 2022Why So Pessimistic? Estimating Uncertainties for Offline RL through Ensembles, and Why Their Independence Matters27 May
  5. 2022A Survey on Offline Reinforcement Learning: Taxonomy, Review, and Open Problems2 Mar
  6. 2022Supported Policy Optimization for Offline Reinforcement Learning13 Feb
  7. 2021RvS: What is Essential for Offline RL via Supervised Learning?20 Dec
  8. 2021Offline Reinforcement Learning with Implicit Q-Learning12 Oct
  9. 2021A Minimalist Approach to Offline Reinforcement Learning12 Jun
  10. 2020PLAS: Latent Action Space for Offline Reinforcement Learning14 Nov
  11. 2020Conservative Q-Learning for Offline Reinforcement Learning8 Jun
  12. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems4 May
  13. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learning15 Apr

Other references · 22

  1. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  2. 2017Proximal Policy Optimization Algorithmscited by 14,441
  3. 2015Continuous control with deep reinforcement learningcited by 3,279
  4. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,302
  5. 2021Decision Transformer: Reinforcement Learning via Sequence Modelingcited by 1,528
  6. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,227
  7. 2019Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reductioncited by 833
  8. 2019Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learningcited by 750
  9. 2020AWAC: Accelerating Online Reinforcement Learning with Offline Datasetscited by 647
  10. 2022Diffusion Policies as an Expressive Policy Class for Offline Reinforcement Learningcited by 543
  11. 2023Q-Transformer: Scalable Offline Reinforcement Learning via Autoregressive Q-Functionscited by 129
  12. 2022Q-learning Decision Transformer: Leveraging Dynamic Programming for Conditional Sequence Modelling in Offline RLcited by 103
10 more
  1. 2019BAIL: Best-Action Imitation Learning for Batch Deep Reinforcement Learningcited by 90
  2. 2022CIRS: Bursting Filter Bubbles by Counterfactual Interactive Recommender Systemcited by 38
  3. 2023Emergent Agentic Transformer from Chain of Hindsight Experiencecited by 33
  4. 2023ACT: Empowering Decision Transformer with Dynamic Programming via Advantage Conditioningcited by 24
  5. 2022Contrastive Identity-Aware Learning for Multi-Agent Value Decompositioncited by 19
  6. 2023Is Centralized Training with Decentralized Execution Framework Centralized Enough for MARL?cited by 18
  7. 2022Learning from Good Trajectories in Offline Multi-Agent Reinforcement Learningcited by 12
  8. 2023Curricular Subgoals for Inverse Reinforcement Learningcited by 5
  9. 2024Powerformer: A Section-adaptive Transformer for Power Flow Adjustmentcited by 5
  10. 2022Fast Offline Policy Optimization for Large Scale Recommendationcited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 5; 2026: 5.

10 papers.

  1. 2025Active Advantage-Aligned Online Reinforcement Learning with Offline Datacited by 5
  2. 2025Supervised Optimism Correction: Be Confident When LLMs Are Surecited by 5
  3. 2025BiTrajDiff: Bidirectional Trajectory Generation with Diffusion Models for Offline Reinforcement Learningcited by 4
  4. 2025Sequential Multi-Agent Dynamic Algorithm Configurationcited by 3
  5. 2026ExToken: Structured Exploration for Efficient Vision-Language-Action Reinforcement Fine-tuningcited by 2
  6. 2025CAWR: Corruption-Averse Advantage-Weighted Regression for Robust Policy Optimizationcited by 0
  7. 2026Language-Critique Imitation Learning from Suboptimal Demonstrationscited by 0
  8. 2026Improving Online Reinforcement Learning via Bidirectional Behavior Prior Distillationcited by 0
  9. 2026PF-RL: Progress Field Reinforcement Learning via Goal-Conditioned Value Geometry for Vision-Language-Action Modelscited by 0
  10. 2026Policy Plasticity Matters in Offline-to-Online Reinforcement Learning: Refitting Offline Policies for Online Adaptationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Adaptive Advantage-Guided Policy Regularization for Offline Reinforcement Learning
  2. 2025Taming OOD Actions for Offline Reinforcement Learning: An Advantage-Based Approach
  3. 2022A Unified Framework for Alternating Offline Model Training and Policy Learning
  4. 2022Latent-Variable Advantage-Weighted Policy Optimization for Offline RL
  5. 2022Offline Reinforcement Learning with Closed-Form Policy Improvement Operators
  6. 2022DARA: Dynamics-Aware Reward Augmentation in Offline Reinforcement Learning
  7. 2024Offline Multi-Agent Reinforcement Learning via In-Sample Sequential Policy Optimization
  8. 2023Boosting Offline Reinforcement Learning with Action Preference Query