mlsanityData to 7 Oct 2026

Multi Task Inverse Reinforcement Learning for Common Sense Reward

arXiv 2402.11367 ↗First posted 17 Feb 2024cs.LGcited by 0

On the map: Offline RL and imitation learning from data → Inverse reinforcement learning

Abstract (arXiv)

One of the challenges in applying reinforcement learning in a complex real-world environment lies in providing the agent with a sufficiently detailed reward function. Any misalignment between the reward and the desired behavior can result in unwanted outcomes. This may lead to issues like "reward hacking" where the agent maximizes rewards by unintended behavior. In this work, we propose to disentangle the reward into two distinct parts. A simple task-specific reward, outlining the particulars of the task at hand, and an unknown common-sense reward, indicating the expected behavior of the agent within the environment. We then explore how this common-sense reward can be learned from expert demonstrations. We first show that inverse reinforcement learning, even when it succeeds in training an agent, does not learn a useful reward function. That is, training a new agent with the learned reward does not impair the desired behaviors. We then demonstrate that this problem can be solved by training simultaneously on multiple tasks. That is, multi-task inverse reinforcement learning can be applied to learn a useful reward function.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Inverse reinforcement learning, one of 13 directions in Offline RL and imitation learning from data. Builds on 32 of its 45 references that are in the map. First posted on arXiv 17 Feb 2024.

Builds on

32 of 45 references.

Closest ancestors · 7

Same direction, newest first

  1. 2023Multi-task Hierarchical Adversarial Inverse Reinforcement Learning22 May
  2. 2022Robust Imitation via Mirror Descent Inverse Reinforcement Learning20 Oct
  3. 2020Regularized Inverse Reinforcement Learning7 Oct
  4. 2019Meta-Inverse Reinforcement Learning with Probabilistic Context Variables20 Sep
  5. 2018A Survey of Inverse Reinforcement Learning: Challenges, Methods and Progress18 Jun
  6. 2018Learning a Prior over Intent via Meta-Inverse Reinforcement Learning31 May
  7. 2018Multi-task Maximum Entropy Inverse Reinforcement Learning22 May

Other references · 25

  1. 2017Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networkscited by 4,861
  2. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  3. 2016Generative Adversarial Imitation Learningcited by 1,576
  4. 2020Gradient Surgery for Multi-Task Learningcited by 1,210
  5. 2019Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learningcited by 1,168
  6. 2017An Overview of Multi-Task Learning in Deep Neural Networkscited by 791
  7. 2021Conflict-Averse Gradient Descent for Multi-task Learningcited by 441
  8. 2019Efficient Off-Policy Meta-Reinforcement Learning via Probabilistic Context Variablescited by 372
  9. 2017One-Shot Visual Imitation Learning via Meta-Learningcited by 225
  10. 2022Multi-Task Learning as a Bargaining Gamecited by 209
  11. 2021Multi-Task Reinforcement Learning with Context-based Representationscited by 169
  12. 2016A Connection between Generative Adversarial Networks, Inverse Reinforcement Learning, and Energy-Based Modelscited by 167
13 more
  1. 2018One-Shot Imitation from Observing Humans via Domain-Adaptive Meta-Learningcited by 146
  2. 2020Multi-Task Reinforcement Learning with Soft Modularizationcited by 138
  3. 2018Variational Discriminator Bottleneck: Improving Imitation Learning, Inverse RL, and GANs by Constraining Information Flowcited by 103
  4. 2021Conservative Data Sharing for Multi-Task Offline Reinforcement Learningcited by 64
  5. 2020Auxiliary Learning by Implicit Differentiationcited by 45
  6. 2020Inverse Constrained Reinforcement Learningcited by 45
  7. 2022imitation: Clean Imitation Learning Implementationscited by 34
  8. 2022On Implicit Bias in Overparameterized Bilevel Optimizationcited by 30
  9. 2020Joint Goal and Strategy Inference across Heterogeneous Demonstrators via Reward Network Distillationcited by 19
  10. 2023Learning Safety Constraints from Demonstrations with Unknown Rewardscited by 19
  11. 2023Learning Shared Safety Constraints from Multi-task Demonstrationscited by 15
  12. 2023QMP: Q-switch Mixture of Policies for Multi-Task Behavior Sharingcited by 6
  13. 2020Maximum Entropy Multi-Task Inverse RLcited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2021Versatile Inverse Reinforcement Learning via Cumulative Rewards
  2. 2026Scalable Multi-Task Inverse Reinforcement Learning
  3. 2026Generalize and Guide: Decomposing Rewards for Few-Shot Inverse Reinforcement Learning
  4. 2025Generalizing Behavior via Inverse Reinforcement Learning with Closed-Form Reward Centroids
  5. 2023BC-IRL: Learning Generalizable Reward Functions from Demonstrations
  6. 2025Inversely Learning Transferable Rewards via Abstracted States
  7. 2024Inverse Reinforcement Learning with Multiple Planning Horizons
  8. 2024Rethinking Inverse Reinforcement Learning: from Data Alignment to Task Alignment