mlsanityData to 7 Oct 2026

MAHALO: Unifying Offline Reinforcement Learning and Imitation Learning from Observations

arXiv 2303.17156 ↗First posted 30 Mar 2023cs.LGcited by 18

On the map: Offline RL and imitation learning from data → Imitation learning methods

Abstract (arXiv)

We study a new paradigm for sequential decision making, called offline policy learning from observations (PLfO). Offline PLfO aims to learn policies using datasets with substandard qualities: 1) only a subset of trajectories is labeled with rewards, 2) labeled trajectories may not contain actions, 3) labeled trajectories may not be of high quality, and 4) the data may not have full coverage. Such imperfection is common in real-world learning scenarios, and offline PLfO encompasses many existing offline learning setups, including offline imitation learning (IL), offline IL from observations (ILfO), and offline reinforcement learning (RL). In this work, we present a generic approach to offline PLfO, called Modality-agnostic Adversarial Hypothesis Adaptation for Learning from Observations (MAHALO). Built upon the pessimism concept in offline RL, MAHALO optimizes the policy using a performance lower bound that accounts for uncertainty due to the dataset's insufficient coverage. We implement this idea by adversarially training data-consistent critic and reward functions, which forces the learned policy to be robust to data deficiency. We show that MAHALO consistently outperforms or matches specialized algorithms across a variety of offline PLfO tasks in theory and experiments. Our code is available at https://github.com/AnqiLi/mahalo.

Key facts. Cited by 18 papers in the map (data through 7 Oct 2026). Placed in Imitation learning methods, one of 13 directions in Offline RL and imitation learning from data. Builds on 49 of its 70 references that are in the map. First posted on arXiv 30 Mar 2023.

Builds on

49 of 70 references.

Closest ancestors · 13

Same direction, newest first

  1. 2023A Strong Baseline for Batch Imitation Learning6 Feb
  2. 2022Discriminator-Weighted Offline Imitation Learning from Suboptimal Demonstrations20 Jul
  3. 2022Versatile Offline Imitation from Observations and Examples via Regularized State-Occupancy Matching4 Feb
  4. 2021Learning Feasibility to Imitate Demonstrators with Different Dynamics28 Oct
  5. 2021Learning from Imperfect Demonstrations from Agents with Varying Dynamics10 Mar
  6. 2021Off-Policy Imitation Learning from Observations25 Feb
  7. 2021MobILE: Model-Based Imitation Learning From Observation Alone22 Feb
  8. 2020Offline Learning from Demonstrations and Unlabeled Experience27 Nov
  9. 2019State Alignment-based Imitation Learning21 Nov
  10. 2019Imitation Learning from Observations by Minimizing Inverse Dynamics Disagreement10 Oct
  11. 2019Recent Advances in Imitation Learning from Observation30 May
  12. 2019Random Expert Distillation: Imitation Learning via Expert Policy Support Estimation16 May
1 more
  1. 2018Behavioral Cloning from Observation4 May

Other references · 36

  1. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  2. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  3. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,302
  4. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problemscited by 1,873
  5. 2020Conservative Q-Learning for Offline Reinforcement Learningcited by 1,777
  6. 2016Generative Adversarial Imitation Learningcited by 1,576
  7. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learningcited by 1,406
  8. 2021Offline Reinforcement Learning with Implicit Q-Learningcited by 1,245
  9. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,227
  10. 2019Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learningcited by 1,168
  11. 2021A Minimalist Approach to Offline Reinforcement Learningcited by 882
  12. 2019Behavior Regularized Offline Reinforcement Learningcited by 654
24 more
  1. 2020MOPO: Model-based Offline Policy Optimizationcited by 630
  2. 2020MOReL : Model-Based Offline Reinforcement Learningcited by 582
  3. 2021COMBO: Conservative Offline Model-Based Policy Optimizationcited by 365
  4. 2020Is Pessimism Provably Efficient for Offline RL?cited by 362
  5. 2021Bellman-consistent Pessimism for Offline Reinforcement Learningcited by 267
  6. 2021How to Train Your Robot with Deep Reinforcement Learning; Lessons We've Learnedcited by 201
  7. 2021Pessimistic Model-based Offline Reinforcement Learning under Partial Coveragecited by 160
  8. 2017Safe Policy Improvement with Baseline Bootstrappingcited by 141
  9. 2019End-to-End Robotic Reinforcement Learning without Reward Engineeringcited by 115
  10. 2022RAMBO-RL: Robust Adversarial Model-Based Offline Reinforcement Learningcited by 115
  11. 2022Adversarially Trained Actor Critic for Offline Reinforcement Learningcited by 111
  12. 2020State-Only Imitation Learning for Dexterous Manipulationcited by 90
  13. 2020COG: Connecting New Skills to Past Experience with Offline Reinforcement Learningcited by 83
  14. 2020Reinforcement Learning with Videos: Combining Offline Observations with Interactioncited by 80
  15. 2018Variational Inverse Control with Events: A General Framework for Data-Driven Reward Definitioncited by 66
  16. 2022How to Leverage Unlabeled Data in Offline Reinforcement Learningcited by 61
  17. 2018Evaluating Reinforcement Learning Algorithms in Observational Health Settingscited by 60
  18. 2021Replacing Rewards with Examples: Example-Based Policy Search via Recursive Classificationcited by 38
  19. 2020Semi-supervised reward learning for offline reinforcement learningcited by 22
  20. 2023CLARE: Conservative Model-Based Reward Learning for Offline Inverse Reinforcement Learningcited by 20
  21. 2023Mind the Gap: Offline Policy Optimization for Imperfect Rewardscited by 15
  22. 2020Estimating Q(s,s') with Deep Deterministic Dynamics Gradientscited by 12
  23. 2023The Provable Benefits of Unsupervised Data Sharing for Offline Reinforcement Learningcited by 10
  24. 2022ARMOR: A Model-based Framework for Improving Arbitrary Baseline Policies with Offline Datacited by 9

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 3; 2024: 7; 2025: 6; 2026: 2.

18 papers.

Sort:

By area:

  1. 2023A Survey of Imitation Learning: Algorithms, Recent Developments, and Challengescited by 145
  2. 2023Survival Instinct in Offline Reinforcement Learningcited by 20
  3. 2024SEABO: A Simple Search-Based Method for Offline Imitation Learningcited by 14
  4. 2024Robot Policy Learning with Temporal Optimal Transport Rewardcited by 8
  5. 2024Hybrid Reinforcement Learning from Offline Observation Alonecited by 5
  6. 2025Self-evolved Imitation Learning in Simulated Worldcited by 4
  7. 2023A Simple Solution for Offline Imitation from Observations and Examples with Possibly Incomplete Trajectoriescited by 3
  8. 2024Overcoming Knowledge Barriers: Online Imitation Learning from Visual Observation with Pretrained World Modelscited by 2
  9. 2024Dynamic Non-Prehensile Object Transport via Model-Predictive Reinforcement Learningcited by 2
  10. 2024How to Solve Contextual Goal-Oriented Problems with Offline Datasets?cited by 1
  11. 2025Video-Enhanced Offline Reinforcement Learning: A Model-Based Approachcited by 1
  12. 2025OffSim: Offline Simulator for Model-based Offline Inverse Reinforcement Learningcited by 1
6 more
  1. 2024Augmenting Offline RL with Unlabeled Datacited by 0
  2. 2025TROFI: Trajectory-Ranked Offline Inverse Reinforcement Learningcited by 0
  3. 2025Learning from Observation: A Survey of Recent Advancescited by 0
  4. 2025Quantifying Generalisation in Imitation Learningcited by 0
  5. 2026Off-Policy Evaluation for Missingness-Aware Policies in MDPs with Rewards Missing Not at Randomcited by 0
  6. 2026Reward as Observation: Learning Reward-Based Policies for Rapid Adaptationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2022A Policy-Guided Imitation Approach for Offline Reinforcement Learning
  2. 2021Representation Matters: Offline Pretraining for Sequential Decision Making
  3. 2023Beyond Uniform Sampling: Offline Reinforcement Learning with Imbalanced Datasets
  4. 2021Bridging Offline Reinforcement Learning and Imitation Learning: A Tale of Pessimism
  5. 2023Offline RL with Observation Histories: Analyzing and Improving Sample Complexity
  6. 2024A Dual Approach to Imitation Learning from Observations with Offline Datasets
  7. 2024Offline Imitation Learning by Controlling the Effective Planning Horizon
  8. 2023Using Offline Data to Speed Up Reinforcement Learning in Procedurally Generated Environments