mlsanityData to 7 Oct 2026

PAGAR: Taming Reward Misalignment in Inverse Reinforcement Learning-Based Imitation Learning with Protagonist Antagonist Guided Adversarial Reward

arXiv 2306.01731 ↗First posted 2 Jun 2023cs.LGcited by 0

On the map: Offline RL and imitation learning from data → Imitation learning methods

Abstract (arXiv)

Many imitation learning (IL) algorithms employ inverse reinforcement learning (IRL) to infer the intrinsic reward function that an expert is implicitly optimizing for based on their demonstrated behaviors. However, in practice, IRL-based IL can fail to accomplish the underlying task due to a misalignment between the inferred reward and the objective of the task. In this paper, we address the susceptibility of IL to such misalignment by introducing a semi-supervised reward design paradigm called Protagonist Antagonist Guided Adversarial Reward (PAGAR). PAGAR-based IL trains a policy to perform well under mixed reward functions instead of a single reward function as in IRL-based IL. We identify the theoretical conditions under which PAGAR-based IL can avoid the task failures caused by reward misalignment. We also present a practical on-and-off policy approach to implementing PAGAR-based IL. Experimental results show that our algorithm outperforms standard IL baselines in complex tasks and challenging transfer settings.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Imitation learning methods, one of 13 directions in Offline RL and imitation learning from data. Builds on 25 of its 46 references that are in the map. First posted on arXiv 2 Jun 2023.

Builds on

25 of 46 references.

Closest ancestors · 2

Same direction, newest first

  1. 2021IQ-Learn: Inverse soft-Q Learning for Imitation23 Jun
  2. 2018Variational Discriminator Bottleneck: Improving Imitation Learning, Inverse RL, and GANs by Constraining Information Flow1 Oct

Other references · 23

  1. 2017Proximal Policy Optimization Algorithmscited by 14,441
  2. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  3. 2015Trust Region Policy Optimizationcited by 2,503
  4. 2016Concrete Problems in AI Safetycited by 1,582
  5. 2016Generative Adversarial Imitation Learningcited by 1,576
  6. 2016Guided Cost Learning: Deep Inverse Optimal Control via Policy Optimizationcited by 400
  7. 2022The Effects of Reward Misspecification: Mapping and Mitigating Misaligned Modelscited by 312
  8. 2020Emergent Complexity and Zero-shot Transfer via Unsupervised Environment Designcited by 260
  9. 2019Extrapolating Beyond Suboptimal Demonstrations via Inverse Reinforcement Learning from Observationscited by 256
  10. 2017Inverse Reward Designcited by 246
  11. 2019Reinforcement Learning for Temporal Logic Control Synthesis with Probabilistic Satisfaction Guaranteescited by 77
  12. 2023Inverse Preference Learning: Preference-based RL without a Reward Functioncited by 68
11 more
  1. 2021Identifiability in inverse reinforcement learningcited by 55
  2. 2019On the Feasibility of Learning, Rather than Assuming, Human Biases for Reward Inferencecited by 49
  3. 2022Invariance in Policy Optimisation and Partial Identifiability in Reward Learningcited by 46
  4. 2022Active Exploration for Inverse Reinforcement Learningcited by 27
  5. 2022Misspecification in Inverse Reinforcement Learningcited by 27
  6. 2022On the Sensitivity of Reward Inference to Misspecified Human Modelscited by 25
  7. 2016Generalizing Skills with Semi-Supervised Reinforcement Learningcited by 18
  8. 2017Safety-Aware Apprenticeship Learningcited by 12
  9. 2021Programmatic Reward Design by Examplecited by 9
  10. 2022A Hierarchical Bayesian Approach to Inverse Reinforcement Learning with Symbolic Reward Machinescited by 9
  11. 2020Runtime-Safety-Guided Policy Repaircited by 6

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2024Rethinking Inverse Reinforcement Learning: from Data Alignment to Task Alignment
  2. 2026Provably Efficient Policy-Reward Co-Pretraining for Adversarial Imitation Learning
  3. 2025Trust Region Reward Optimization and Proximal Inverse Reward Optimization Algorithm
  4. 2021Reward function shape exploration in adversarial imitation learning: an empirical study
  5. 2024Rethinking Adversarial Inverse Reinforcement Learning: Policy Imitation, Transferable Reward Recovery and Algebraic Equilibrium Proof
  6. 2024RILe: Reinforced Imitation Learning
  7. 2019Random Expert Distillation: Imitation Learning via Expert Policy Support Estimation
  8. 2025On Discovering Algorithms for Adversarial Imitation Learning