mlsanityData to 14 Oct 2026

PsiPhi-Learning: Reinforcement Learning with Demonstrations using Successor Features and Inverse Temporal Difference Learning

arXiv 2102.12560 ↗First posted 24 Feb 2021cs.LG cs.AIcited by 23

Not on the map. Why?

Abstract (arXiv)

We study reinforcement learning (RL) with no-reward demonstrations, a setting in which an RL agent has access to additional data from the interaction of other agents with the same environment. However, it has no access to the rewards or goals of these agents, and their objectives and levels of expertise may vary widely. These assumptions are common in multi-agent settings, such as autonomous driving. To effectively use this data, we turn to the framework of successor features. This allows us to disentangle shared features and dynamics of the environment from agent-specific rewards and policies. We propose a multi-task inverse reinforcement learning (IRL) algorithm, called inverse temporal difference learning (ITD), that learns shared state features, alongside per-agent successor features and preference vectors, purely from demonstrations without reward labels. We further show how to seamlessly integrate ITD with learning from online environment interactions, arriving at a novel algorithm for reinforcement learning with demonstrations, called Psi Phi-learning (pronounced `Sci-Fi'). We provide empirical evidence for the effectiveness of Psi Phi-learning as a method for improving RL, IRL, imitation, and few-shot transfer, and derive worst-case bounds for its performance in zero-shot transfer to new tasks.

Key facts. Cited by 23 papers in the map (data through 14 Oct 2026). Builds on 47 of its 99 references that are in the map. First posted on arXiv 24 Feb 2021.

Builds on

47 of 99 references.

References · 47

  1. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  2. 2017Proximal Policy Optimization Algorithmscited by 15,215
  3. 2013Playing Atari with Deep Reinforcement Learningcited by 3,274
  4. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problemscited by 1,920
  5. 2016Generative Adversarial Imitation Learningcited by 1,597
  6. 2018IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architecturescited by 883
  7. 2018QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulationcited by 743
  8. 2017Learning Complex Dexterous Manipulation with Deep Reinforcement Learning and Demonstrationscited by 715
  9. 2018Behavioral Cloning from Observationcited by 492
  10. 2017Learning Robust Rewards with Adversarial Inverse Reinforcement Learningcited by 490
  11. 2017End-to-end Driving via Conditional Imitation Learningcited by 485
  12. 2019Leveraging Procedural Generation to Benchmark Reinforcement Learningcited by 483
35 more
  1. 2017Time-Contrastive Networks: Self-Supervised Learning from Videocited by 459
  2. 2016Guided Cost Learning: Deep Inverse Optimal Control via Policy Optimizationcited by 406
  3. 2018Tune: A Research Platform for Distributed Model Selection and Trainingcited by 395
  4. 2017Population Based Training of Neural Networkscited by 367
  5. 2016Successor Features for Transfer in Reinforcement Learningcited by 340
  6. 2017Overcoming Exploration in Reinforcement Learning with Demonstrationscited by 340
  7. 2017Leveraging Demonstrations for Deep Reinforcement Learning on Robotics Problems with Sparse Rewardscited by 319
  8. 2017Deep Imitation Learning for Complex Manipulation Tasks from Virtual Reality Teleoperationcited by 289
  9. 2019Extrapolating Beyond Suboptimal Demonstrations via Inverse Reinforcement Learning from Observationscited by 260
  10. 2018Machine Theory of Mindcited by 226
  11. 2015Maximum Entropy Deep Inverse Reinforcement Learningcited by 176
  12. 2016A Connection between Generative Adversarial Networks, Inverse Reinforcement Learning, and Energy-Based Modelscited by 168
  13. 2017Imitation from Observation: Learning to Imitate Behaviors from Raw Video via Context Translationcited by 167
  14. 2020Acme: A Research Framework for Distributed Reinforcement Learningcited by 146
  15. 2020Can Autonomous Vehicles Identify, Recover From, and Adapt to Distribution Shifts?cited by 139
  16. 2019Fast Task Inference with Variational Intrinsic Successor Featurescited by 133
  17. 2017Autonomous Agents Modelling Other Agents: A Comprehensive Survey and Open Problemscited by 131
  18. 2018Count-Based Exploration with the Successor Representationcited by 112
  19. 2018Universal Successor Features Approximatorscited by 103
  20. 2017Vision-Based Multi-Task Manipulation for Inexpensive Robots Using End-To-End Learning from Demonstrationcited by 100
  21. 2018Playing hard exploration games by watching YouTubecited by 99
  22. 2016Opponent Modeling in Deep Reinforcement Learningcited by 97
  23. 2017A Survey of Learning in Multiagent Environments: Dealing with Non-Stationaritycited by 94
  24. 2017Eigenoption Discovery through the Deep Successor Representationcited by 86
  25. 2018Deep Imitative Models for Flexible Inference, Planning, and Controlcited by 76
  26. 2018Multiple Interactions Made Easy (MIME): Large Scale Demonstrations Data for Imitationcited by 76
  27. 2018Reinforcement Learning from Imperfect Demonstrationscited by 75
  28. 2016Deep Successor Reinforcement Learningcited by 72
  29. 2019From Language to Goals: Inverse Reinforcement Learning for Vision-Based Instruction Followingcited by 67
  30. 2019Making Efficient Use of Demonstrations to Solve Hard Exploration Problemscited by 45
  31. 2018Successor Uncertainties: Exploration and Uncertainty in Temporal Difference Learningcited by 35
  32. 2019Truly Batch Apprenticeship Learning with Deep Successor Featurescited by 27
  33. 2019Agent Modeling as Auxiliary Task for Deep Reinforcement Learningcited by 14
  34. 2017Observational Learning by Reinforcement Learningcited by 12
  35. 2018One-Shot High-Fidelity Imitation: Training Large-Scale Deep Nets with RLcited by 11

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 2; 2022: 5; 2023: 4; 2024: 4; 2025: 3; 2026: 5.

23 papers.

Sort:

By area:

  1. 2022Learning from All Vehiclescited by 135
  2. 2022Contrastive Value Learning: Implicit Models for Simple Offline RLcited by 15
  3. 2022CACTO: Continuous Actor-Critic with Trajectory Optimization -- Towards global optimalitycited by 12
  4. 2023Combining Behaviors with the Successor Features Keyboardcited by 12
  5. 2023Self-Supervised Reinforcement Learning that Transfers using Random Featurescited by 11
  6. 2024Learning Successor Features the Simple Waycited by 10
  7. 2024Non-Adversarial Inverse Reinforcement Learning via Successor Feature Matchingcited by 10
  8. 2021Self-Consistent Models and Valuescited by 7
  9. 2023Composing Task Knowledge with Modular Successor Feature Approximatorscited by 7
  10. 2021Model-Value Inconsistency as a Signal for Epistemic Uncertaintycited by 6
  11. 2022Basis for Intentions: Efficient Inverse Reinforcement Learning using Past Experiencecited by 4
  12. 2024Towards Sample-Efficiency and Generalization of Transfer and Inverse Reinforcement Learning: A Comprehensive Literature Reviewcited by 3
11 more
  1. 2025SR-Reward: Taking The Path More Traveledcited by 3
  2. 2025An Efficient Open World Environment for Multi-Agent Social Learningcited by 3
  3. 2024A Bayesian Solution To The Imitation Gapcited by 2
  4. 2022Task Relabelling for Multi-task Transfer using Successor Featurescited by 1
  5. 2023Uncertainty-aware transfer across tasks using hybrid model-based successor feature reinforcement learningcited by 1
  6. 2025Cross-cultural Deployment of Autonomous Vehicles Using Data-light Inverse Reinforcement Learningcited by 0
  7. 2026Exploiting Expertise of Non-Expert and Diverse Agents in Social Bandit Learning: A Free Energy Approachcited by 0
  8. 2026Full-Gradient Successor Feature Representationscited by 0
  9. 2026ConTraIRL: Factorized Contrastive Abstractions for Transferable IRLcited by 0
  10. 2026Do as the Romans Do: Learning Universal Behaviors from Heterogeneous Agentscited by 0
  11. 2026Generalize and Guide: Decomposing Rewards for Few-Shot Inverse Reinforcement Learningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2019Meta-Inverse Reinforcement Learning with Probabilistic Context Variables
  2. 2018A Survey of Inverse Reinforcement Learning: Challenges, Methods and Progress
  3. 2024Multi Task Inverse Reinforcement Learning for Common Sense Reward
  4. 2018Machine Teaching for Inverse Reinforcement Learning: Algorithms and Applications
  5. 2023Is Inverse Reinforcement Learning Harder than Standard Reinforcement Learning? A Theoretical Perspective
  6. 2020Inverse Reinforcement Learning from a Gradient-based Learner
  7. 2018Learning a Prior over Intent via Meta-Inverse Reinforcement Learning
  8. 2019Watch, Try, Learn: Meta-Learning from Demonstrations and Reward