mlsanityData to 7 Oct 2026

Rethinking Inverse Reinforcement Learning: from Data Alignment to Task Alignment

arXiv 2410.23680 ↗First posted 31 Oct 2024cs.LG cs.AIcited by 3

On the map: Offline RL and imitation learning from data → Inverse reinforcement learning

Abstract (arXiv)

Many imitation learning (IL) algorithms use inverse reinforcement learning (IRL) to infer a reward function that aligns with the demonstration. However, the inferred reward functions often fail to capture the underlying task objectives. In this paper, we propose a novel framework for IRL-based IL that prioritizes task alignment over conventional data alignment. Our framework is a semi-supervised approach that leverages expert demonstrations as weak supervision to derive a set of candidate reward functions that align with the task rather than only with the data. It then adopts an adversarial mechanism to train a policy with this set of reward functions to gain a collective validation of the policy's ability to accomplish the task. We provide theoretical insights into this framework's ability to mitigate task-reward misalignment and present a practical implementation. Our experimental results show that our framework outperforms conventional IL baselines in complex and transfer learning scenarios.

Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Placed in Inverse reinforcement learning, one of 13 directions in Offline RL and imitation learning from data. Builds on 26 of its 40 references that are in the map. First posted on arXiv 31 Oct 2024.

Builds on

26 of 40 references.

Closest ancestors · 6

Same direction, newest first

  1. 2022Misspecification in Inverse Reinforcement Learning6 Dec
  2. 2022Active Exploration for Inverse Reinforcement Learning18 Jul
  3. 2022A Hierarchical Bayesian Approach to Inverse Reinforcement Learning with Symbolic Reward Machines20 Apr
  4. 2021Identifiability in inverse reinforcement learning7 Jun
  5. 2020f-IRL: Inverse Reinforcement Learning via State Marginal Matching9 Nov
  6. 2020Robust Inverse Reinforcement Learning under Transition Dynamics Mismatch2 Jul

Other references · 20

  1. 2017Proximal Policy Optimization Algorithmscited by 14,441
  2. 2015Trust Region Policy Optimizationcited by 2,503
  3. 2016Concrete Problems in AI Safetycited by 1,582
  4. 2016Generative Adversarial Imitation Learningcited by 1,576
  5. 2016Guided Cost Learning: Deep Inverse Optimal Control via Policy Optimizationcited by 400
  6. 2023Minigrid & Miniworld: Modular & Customizable Reinforcement Learning Environments for Goal-Oriented Taskscited by 320
  7. 2022The Effects of Reward Misspecification: Mapping and Mitigating Misaligned Modelscited by 312
  8. 2020Emergent Complexity and Zero-shot Transfer via Unsupervised Environment Designcited by 260
  9. 2017Inverse Reward Designcited by 246
  10. 2021IQ-Learn: Inverse soft-Q Learning for Imitationcited by 191
  11. 2018Variational Discriminator Bottleneck: Improving Imitation Learning, Inverse RL, and GANs by Constraining Information Flowcited by 103
  12. 2021On the Expressivity of Markov Rewardcited by 71
8 more
  1. 2023Inverse Preference Learning: Preference-based RL without a Reward Functioncited by 68
  2. 2023Dual RL: Unification and New Methods for Reinforcement and Imitation Learningcited by 48
  3. 2022Invariance in Policy Optimisation and Partial Identifiability in Reward Learningcited by 46
  4. 2022On the Sensitivity of Reward Inference to Misspecified Human Modelscited by 25
  5. 2016Generalizing Skills with Semi-Supervised Reinforcement Learningcited by 18
  6. 2017Safety-Aware Apprenticeship Learningcited by 12
  7. 2021Programmatic Reward Design by Examplecited by 9
  8. 2020Runtime-Safety-Guided Policy Repaircited by 6

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 2; 2026: 1.

3 papers.

  1. 2026Behavioral Mode Discovery for Fine-tuning Multimodal Generative Policiescited by 2
  2. 2025Imitation Learning in the Deep Learning Era: A Novel Taxonomy and Recent Advancescited by 0
  3. 2025Hybrid-AIRL: Enhancing Inverse Reinforcement Learning with Supervised Expert Guidancecited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023PAGAR: Taming Reward Misalignment in Inverse Reinforcement Learning-Based Imitation Learning with Protagonist Antagonist Guided Adversarial Reward
  2. 2021Meta-Adversarial Inverse Reinforcement Learning for Decision-making Tasks
  3. 2025From Demonstrations to Rewards: Alignment Without Explicit Human Preferences
  4. 2026Inverse RL Helps Align AI by Imitating Humans
  5. 2019Meta-Inverse Reinforcement Learning with Probabilistic Context Variables
  6. 2023Adversarial Imitation Learning On Aggregated Data
  7. 2024Inverse Reinforcement Learning by Estimating Expertise of Demonstrators
  8. 2026Interactionless Inverse Reinforcement Learning: A Data-Centric Framework for Durable Alignment