mlsanityData to 7 Oct 2026

Latent-Variable Advantage-Weighted Policy Optimization for Offline RL

arXiv 2203.08949 ↗First posted 16 Mar 2022cs.LGcited by 12

On the map: Offline RL and imitation learning from data → Offline RL algorithms

Abstract (arXiv)

Offline reinforcement learning methods hold the promise of learning policies from pre-collected datasets without the need to query the environment for new transitions. This setting is particularly well-suited for continuous control robotic applications for which online data collection based on trial-and-error is costly and potentially unsafe. In practice, offline datasets are often heterogeneous, i.e., collected in a variety of scenarios, such as data from several human demonstrators or from policies that act with different purposes. Unfortunately, such datasets can exacerbate the distribution shift between the behavior policy underlying the data and the optimal policy to be learned, leading to poor performance. To address this challenge, we propose to leverage latent-variable policies that can represent a broader class of policy distributions, leading to better adherence to the training data distribution while maximizing reward via a policy over the latent variable. As we empirically show on a range of simulated locomotion, navigation, and manipulation tasks, our method referred to as latent-variable advantage-weighted policy optimization (LAPO), improves the average performance of the next best-performing offline reinforcement learning methods by 49% on heterogeneous datasets, and by 8% on datasets with narrow and biased distributions.

Key facts. Cited by 12 papers in the map (data through 7 Oct 2026). Placed in Offline RL algorithms, one of 13 directions in Offline RL and imitation learning from data. Builds on 44 of its 50 references that are in the map. First posted on arXiv 16 Mar 2022.

Builds on

44 of 50 references.

Closest ancestors · 20

Same direction, newest first

  1. 2021Offline Reinforcement Learning with Implicit Q-Learning12 Oct
  2. 2021Conservative Data Sharing for Multi-Task Offline Reinforcement Learning16 Sep
  3. 2021A Minimalist Approach to Offline Reinforcement Learning12 Jun
  4. 2021Offline Reinforcement Learning with Fisher Divergence Critic Regularization14 Mar
  5. 2021S4RL: Surprisingly Simple Self-Supervision for Offline Reinforcement Learning10 Mar
  6. 2021COMBO: Conservative Offline Model-Based Policy Optimization16 Feb
  7. 2021PerSim: Data-Efficient Offline Reinforcement Learning with Heterogeneous Agents via Personalized Simulators13 Feb
  8. 2020Offline Reinforcement Learning from Images with Latent Space Models21 Dec
  9. 2020PLAS: Latent Action Space for Offline Reinforcement Learning14 Nov
  10. 2020OPAL: Offline Primitive Discovery for Accelerating Offline Reinforcement Learning26 Oct
  11. 2020Overcoming Model Bias for Robust Offline Deep Reinforcement Learning12 Aug
  12. 2020EMaQ: Expected-Max Q-Learning Operator for Simple Yet Effective Offline and Online RL21 Jul
8 more
  1. 2020Provably Good Batch Reinforcement Learning Without Great Exploration16 Jul
  2. 2020Conservative Q-Learning for Offline Reinforcement Learning8 Jun
  3. 2020MOPO: Model-based Offline Policy Optimization27 May
  4. 2020MOReL : Model-Based Offline Reinforcement Learning12 May
  5. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems4 May
  6. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learning15 Apr
  7. 2020Keep Doing What Worked: Behavioral Modelling Priors for Offline Reinforcement Learning19 Feb
  8. 2019Behavior Regularized Offline Reinforcement Learning26 Nov

Other references · 24

  1. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  2. 2013Auto-Encoding Variational Bayescited by 3,980
  3. 2015Continuous control with deep reinforcement learningcited by 3,279
  4. 2018Large Scale GAN Training for High Fidelity Natural Image Synthesiscited by 2,609
  5. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,302
  6. 2019Dream to Control: Learning Behaviors by Latent Imaginationcited by 1,598
  7. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,227
  8. 2018Learning Latent Dynamics for Planning from Pixelscited by 1,221
  9. 2019Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reductioncited by 833
  10. 2019Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learningcited by 750
  11. 2020AWAC: Accelerating Online Reinforcement Learning with Offline Datasetscited by 647
  12. 2019Learning Latent Plans from Playcited by 310
12 more
  1. 2019Way Off-Policy Batch Deep Reinforcement Learning of Implicit Human Preferences in Dialogcited by 283
  2. 2019DualDICE: Behavior-Agnostic Estimation of Discounted Stationary Distribution Correctionscited by 254
  3. 2019Stochastic Latent Actor-Critic: Deep Reinforcement Learning with a Latent Variable Modelcited by 228
  4. 2021MT-Opt: Continuous Multi-Task Robotic Reinforcement Learning at Scalecited by 226
  5. 2020Parrot: Data-Driven Behavioral Priors for Reinforcement Learningcited by 136
  6. 2020Model-Based Offline Planningcited by 126
  7. 2020Deployment-Efficient Reinforcement Learning via Model-Based Offline Optimizationcited by 120
  8. 2019MCP: Learning Composable Hierarchical Control with Multiplicative Compositional Policiescited by 105
  9. 2019Meta Reinforcement Learning for Sim-to-real Domain Adaptationcited by 49
  10. 2017Deep Predictive Policy Training using Reinforcement Learningcited by 30
  11. 2021Bayesian Meta-Learning for Few-Shot Policy Adaptation Across Robotic Platformscited by 22
  12. 2020Data-efficient visuomotor policy training using reinforcement learning and generative modelscited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 1; 2023: 4; 2024: 5; 2025: 1; 2026: 1.

12 papers.

  1. 2023State Regularized Policy Optimization on Data with Dynamics Shiftcited by 23
  2. 2023ENOTO: Improving Offline-to-Online Reinforcement Learning with Q-Ensemblescited by 20
  3. 2023Constrained Policy Optimization with Explicit Behavior Density for Offline Reinforcement Learningcited by 16
  4. 2024Adaptive Advantage-Guided Policy Regularization for Offline Reinforcement Learningcited by 13
  5. 2024Rethinking Optimal Transport in Offline Reinforcement Learningcited by 8
  6. 2023Katakomba: Tools and Benchmarks for Data-Driven NetHackcited by 5
  7. 2024Is Value Functions Estimation with Classification Plug-and-play for Offline Reinforcement Learning?cited by 4
  8. 2024Q-Distribution guided Q-learning for offline reinforcement learning: Uncertainty penalized Q-value via consistency modelcited by 4
  9. 2024The Role of Deep Learning Regularizations on Actors in Offline RLcited by 2
  10. 2022Training and Evaluation of Deep Policies using Reinforcement Learning and Generative Modelscited by 1
  11. 2025Diffusion Policies with Value-Conditional Optimization for Offline Reinforcement Learningcited by 1
  12. 2026SPAR: Support-Preserving Action Rectificationcited by 1

Similar, not cited

Close in wording, no citation either way

  1. 2026Latent Policy Steering through One-Step Flow Policies
  2. 2024Constrained Latent Action Policies for Model-Based Offline Reinforcement Learning
  3. 2021A Workflow for Offline Model-Free Robotic Reinforcement Learning
  4. 2023Small Dataset, Big Gains: Enhancing Reinforcement Learning by Offline Pre-Training with Model Based Augmentation
  5. 2023Harnessing Mixed Offline Reinforcement Learning Datasets via Trajectory Weighting
  6. 2025Efficient Reinforcement Learning by Guiding World Models with Non-Curated Data
  7. 2023Reasoning with Latent Diffusion in Offline Reinforcement Learning
  8. 2021Behavioral Priors and Dynamics Models: Improving Performance and Domain Transfer in Offline RL