mlsanityData to 7 Oct 2026

Mimicking Better by Matching the Approximate Action Distribution

arXiv 2306.09805 ↗First posted 16 Jun 2023cs.LGcited by 5

On the map: Offline RL and imitation learning from data → Imitation learning methods

Abstract (arXiv)

In this paper, we introduce MAAD, a novel, sample-efficient on-policy algorithm for Imitation Learning from Observations. MAAD utilizes a surrogate reward signal, which can be derived from various sources such as adversarial games, trajectory matching objectives, or optimal transport criteria. To compensate for the non-availability of expert actions, we rely on an inverse dynamics model that infers plausible actions distribution given the expert's state-state transitions; we regularize the imitator's policy by aligning it to the inferred action distribution. MAAD leads to significantly improved sample efficiency and stability. We demonstrate its effectiveness in a number of MuJoCo environments, both int the OpenAI Gym and the DeepMind Control Suite. We show that it requires considerable fewer interactions to achieve expert performance, outperforming current state-of-the-art on-policy methods. Remarkably, MAAD often stands out as the sole method capable of attaining expert performance levels, underscoring its simplicity and efficacy.

Key facts. Cited by 5 papers in the map (data through 7 Oct 2026). Placed in Imitation learning methods, one of 13 directions in Offline RL and imitation learning from data. Builds on 34 of its 46 references that are in the map. First posted on arXiv 16 Jun 2023.

Builds on

34 of 46 references.

Closest ancestors · 13

Same direction, newest first

  1. 2022Planning for Sample Efficient Imitation Learning18 Oct
  2. 2022Watch and Match: Supercharging Imitation with Regularized Optimal Transport30 Jun
  3. 2021Cross-Domain Imitation Learning via Optimal Transport7 Oct
  4. 2021Off-Policy Imitation Learning from Observations25 Feb
  5. 2020Imitation Learning with Sinkhorn Distances20 Aug
  6. 2020Primal Wasserstein Imitation Learning8 Jun
  7. 2019Imitation Learning via Off-Policy Distribution Matching10 Dec
  8. 2019Imitation Learning from Observations by Minimizing Inverse Dynamics Disagreement10 Oct
  9. 2019Hybrid Reinforcement Learning with Expert State Sequences11 Mar
  10. 2018Discriminator-Actor-Critic: Addressing Sample Inefficiency and Reward Bias in Adversarial Imitation Learning9 Sep
  11. 2018Sample-Efficient Imitation Learning via Generative Adversarial Nets6 Sep
  12. 2018Generative Adversarial Imitation from Observation17 Jul
1 more
  1. 2018Behavioral Cloning from Observation4 May

Other references · 21

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2017Proximal Policy Optimization Algorithmscited by 14,441
  3. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,302
  4. 2013Sinkhorn Distances: Lightspeed Computation of Optimal Transportation Distancescited by 2,277
  5. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problemscited by 1,873
  6. 2016OpenAI Gymcited by 1,684
  7. 2016Generative Adversarial Imitation Learningcited by 1,576
  8. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,227
  9. 2021A Minimalist Approach to Offline Reinforcement Learningcited by 882
  10. 2019The frontier of simulation-based inferencecited by 490
  11. 2020dm_control: Software and Tasks for Continuous Controlcited by 368
  12. 2018Analyzing Inverse Problems with Invertible Neural Networkscited by 239
9 more
  1. 2019Automatic Posterior Transformation for Likelihood-Free Inferencecited by 216
  2. 2017Flexible statistical inference for mechanistic models of neural dynamicscited by 162
  3. 2017Combining Self-Supervised Learning and Imitation for Vision-Based Rope Manipulationcited by 90
  4. 2020State-Only Imitation Learning for Dexterous Manipulationcited by 90
  5. 2019Diagnosing Bottlenecks in Deep Q-learning Algorithmscited by 73
  6. 2023Inverse Dynamics Pretraining Learns Good Representations for Multitask Imitationcited by 30
  7. 2023LS-IQ: Implicit Reward Regularization for Inverse Reinforcement Learningcited by 28
  8. 2019RIDM: Reinforced Inverse Dynamics Modeling for Learning from a Single Observed Demonstrationcited by 19
  9. 2020Stochastic Grounded Action Transformation for Robot Learning in Simulationcited by 16

Built on by

Cited by year

Citing papers per year: 2023: 1; 2024: 0; 2025: 4.

5 papers.

  1. 2023Imitation Learning from Observation with Automatic Discount Schedulingcited by 10
  2. 2025TimeRewarder: Learning Dense Reward from Passive Videos via Frame-wise Temporal Distancecited by 5
  3. 2025Videos are Sample-Efficient Supervisions: Behavior Cloning from Videos via Latent Representationscited by 4
  4. 2025On Generalization and Distributional Update for Mimicking Observations with Adequate Explorationcited by 0
  5. 2025Noise-Guided Transport for Imitation Learningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2019Provably Efficient Imitation Learning from Observation Alone
  2. 2019A New Framework for Query Efficient Active Imitation Learning
  3. 2023Provably Efficient Adversarial Imitation Learning with Unknown Transitions
  4. 2019Random Expert Distillation: Imitation Learning via Expert Policy Support Estimation
  5. 2021A Pragmatic Look at Deep Imitation Learning
  6. 2022Imitation Learning by State-Only Distribution Matching
  7. 2021Imitation by Predicting Observations
  8. 2020State-only Imitation with Transition Dynamics Mismatch