mlsanityData to 7 Oct 2026

A Ranking Game for Imitation Learning

arXiv 2202.03481 ↗First posted 7 Feb 2022cs.LG cs.AI cs.ROcited by 18

On the map: Offline RL and imitation learning from data → Imitation learning methods

Abstract (arXiv)

We propose a new framework for imitation learning -- treating imitation as a two-player ranking-based game between a policy and a reward. In this game, the reward agent learns to satisfy pairwise performance rankings between behaviors, while the policy agent learns to maximize this reward. In imitation learning, near-optimal expert data can be difficult to obtain, and even in the limit of infinite data cannot imply a total ordering over trajectories as preferences can. On the other hand, learning from preferences alone is challenging as a large number of preferences are required to infer a high-dimensional reward function, though preference data is typically much easier to collect than expert demonstrations. The classical inverse reinforcement learning (IRL) formulation learns from expert demonstrations but provides no mechanism to incorporate learning from offline preferences and vice versa. We instantiate the proposed ranking-game framework with a novel ranking loss giving an algorithm that can simultaneously learn from expert demonstrations and preferences, gaining the advantages of both modalities. Our experiments show that the proposed method achieves state-of-the-art sample efficiency and can solve previously unsolvable tasks in the Learning from Observation (LfO) setting. Project video and code can be found at https://hari-sikchi.github.io/rank-game/

Key facts. Cited by 18 papers in the map (data through 7 Oct 2026). Placed in Imitation learning methods, one of 13 directions in Offline RL and imitation learning from data. Builds on 45 of its 90 references that are in the map. First posted on arXiv 7 Feb 2022.

Builds on

45 of 90 references.

Closest ancestors · 18

Same direction, newest first

  1. 2021IQ-Learn: Inverse soft-Q Learning for Imitation23 Jun
  2. 2021What Matters for Adversarial Imitation Learning?1 Jun
  3. 2021Of Moments and Matching: A Game-Theoretic Framework for Closing the Imitation Gap4 Mar
  4. 2021Off-Policy Imitation Learning from Observations25 Feb
  5. 2021MobILE: Model-Based Imitation Learning From Observation Alone22 Feb
  6. 2020Error Bounds of Imitating Policies and Environments22 Oct
  7. 2020Non-Adversarial Imitation Learning and its Connections to Adversarial Methods8 Aug
  8. 2020Safe Imitation Learning via Fast Bayesian Reward Inference from Preferences21 Feb
  9. 2019Imitation Learning via Off-Policy Distribution Matching10 Dec
  10. 2019State Alignment-based Imitation Learning21 Nov
  11. 2019A Divergence Minimization Perspective on Imitation Learning Methods6 Nov
  12. 2019Imitation Learning from Observations by Minimizing Inverse Dynamics Disagreement10 Oct
6 more
  1. 2019Recent Advances in Imitation Learning from Observation30 May
  2. 2019Imitation Learning as f-Divergence Minimization30 May
  3. 2019Provably Efficient Imitation Learning from Observation Alone27 May
  4. 2018Generative Adversarial Imitation from Observation17 Jul
  5. 2018Imitating Latent Policies from Observation21 May
  6. 2018Behavioral Cloning from Observation4 May

Other references · 27

  1. 2017mixup: Beyond Empirical Risk Minimizationcited by 5,155
  2. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  3. 2017Deep reinforcement learning from human preferencescited by 3,846
  4. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problemscited by 1,873
  5. 2016Generative Adversarial Imitation Learningcited by 1,576
  6. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learningcited by 1,406
  7. 2016Least Squares Generative Adversarial Networkscited by 1,000
  8. 2017Deep Reinforcement Learning that Matterscited by 727
  9. 2017Learning Complex Dexterous Manipulation with Deep Reinforcement Learning and Demonstrationscited by 685
  10. 2017Learning Robust Rewards with Adversarial Inverse Reinforcement Learningcited by 485
  11. 2020robosuite: A Modular Simulation Framework and Benchmark for Robot Learningcited by 484
  12. 2016Guided Cost Learning: Deep Inverse Optimal Control via Policy Optimizationcited by 400
15 more
  1. 2018Reward learning from human preferences and demonstrations in Ataricited by 314
  2. 2019Extrapolating Beyond Suboptimal Demonstrations via Inverse Reinforcement Learning from Observationscited by 256
  3. 2018MixUp as Locally Linear Out-Of-Manifold Regularizationcited by 158
  4. 2020Reward-rational (implicit) choice: A unifying formalism for reward learningcited by 119
  5. 2020Learning from Suboptimal Demonstration via Self-Supervised Reward Regressioncited by 84
  6. 2020A Game Theoretic Framework for Model Based Reinforcement Learningcited by 72
  7. 2020Learning Reward Functions from Diverse Sources of Human Feedback: Optimally Integrating Demonstrations and Preferencescited by 71
  8. 2020f-IRL: Inverse Reinforcement Learning via State Marginal Matchingcited by 67
  9. 2019Learning Reward Functions by Integrating Human Demonstrations and Preferencescited by 64
  10. 2015Increasing the Action Gap: New Operators for Reinforcement Learningcited by 49
  11. 2020When Humans Aren't Optimal: Robots that Collaborate with Risk-Aware Humanscited by 34
  12. 2019Competitive Gradient Descentcited by 32
  13. 2019Convergence of Learning Dynamics in Stackelberg Gamescited by 28
  14. 2019Reinforcement Learning from Imperfect Demonstrations under Soft Expert Guidancecited by 25
  15. 2021Stackelberg Actor-Critic: Game-Theoretic Reinforcement Learning Algorithmscited by 18

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 4; 2024: 9; 2025: 4; 2026: 1.

18 papers.

Sort:

By area:

  1. 2024A Minimaximalist Approach to Reinforcement Learning from Human Feedbackcited by 141
  2. 2024Scaling Laws for Reward Model Overoptimization in Direct Alignment Algorithmscited by 139
  3. 2023Contrastive Preference Learning: Learning from Human Feedback without RLcited by 77
  4. 2023Dual RL: Unification and New Methods for Reinforcement and Imitation Learningcited by 48
  5. 2024Aligning Language Models with Demonstrated Feedbackcited by 32
  6. 2023LS-IQ: Implicit Reward Regularization for Inverse Reinforcement Learningcited by 28
  7. 2023SMORE: Score Models for Offline Goal-Conditioned Reinforcement Learningcited by 21
  8. 2024Towards Principled Representation Learning from Videos for Reinforcement Learningcited by 10
  9. 2024A Dual Approach to Imitation Learning from Observations with Offline Datasetscited by 10
  10. 2024Sample-Efficient Preference-based Reinforcement Learning with Dynamics Aware Rewardscited by 9
  11. 2025CREStE: Scalable Mapless Navigation with Internet Scale Priors and Counterfactual Guidancecited by 9
  12. 2025IL-SOAR : Imitation Learning with Soft Optimistic Actor cRiticcited by 5
6 more
  1. 2024Robot Air Hockey: A Manipulation Testbed for Robot Learning with Reinforcement Learningcited by 4
  2. 2024A Unified Linear Programming Framework for Offline Reward Learning from Human Demonstrations and Feedbackcited by 4
  3. 2024Expert Proximity as Surrogate Rewards for Single Demonstration Imitation Learningcited by 2
  4. 2025On Generalization and Distributional Update for Mimicking Observations with Adequate Explorationcited by 0
  5. 2025Learning from Observation: A Survey of Recent Advancescited by 0
  6. 2026Inverse Reinforcement Learning without an Optimal Demonstrator: A Feasible Reward Set Approachcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2019Better-than-Demonstrator Imitation Learning via Automatically-Ranked Demonstrations
  2. 2021Imitation Learning by Reinforcement Learning
  3. 2024Imitation Learning from Suboptimal Demonstrations via Meta-Learning An Action Ranker
  4. 2021Hyperparameter Selection for Imitation Learning
  5. 2025PROF: An LLM-based Reward Code Preference Optimization Framework for Offline Imitation Learning
  6. 2026Matching Multiple Experts: On the Exploitability of Multi-Agent Imitation Learning
  7. 2022Imitation Learning by Estimating Expertise of Demonstrators
  8. 2019Reinforced Imitation in Heterogeneous Action Space