mlsanityData to 7 Oct 2026

Proximal Point Imitation Learning

arXiv 2209.10968 ↗First posted 22 Sep 2022cs.LGcited by 25

On the map: Offline RL and imitation learning from data → Imitation learning methods

Abstract (arXiv)

This work develops new algorithms with rigorous efficiency guarantees for infinite horizon imitation learning (IL) with linear function approximation without restrictive coherence assumptions. We begin with the minimax formulation of the problem and then outline how to leverage classical tools from optimization, in particular, the proximal-point method (PPM) and dual smoothing, for online and offline IL, respectively. Thanks to PPM, we avoid nested policy evaluation and cost updates for online IL appearing in the prior literature. In particular, we do away with the conventional alternating updates by the optimization of a single convex and smooth objective over both cost and Q-functions. When solved inexactly, we relate the optimization errors to the suboptimality of the recovered policy. As an added bonus, by re-interpreting PPM as dual smoothing with the expert policy as a center point, we also obtain an offline IL algorithm enjoying theoretical guarantees in terms of required expert trajectories. Finally, we achieve convincing empirical performance for both linear and neural network function approximation.

Key facts. Cited by 25 papers in the map (data through 7 Oct 2026). Placed in Imitation learning methods, one of 13 directions in Offline RL and imitation learning from data. Builds on 58 of its 138 references that are in the map. First posted on arXiv 22 Sep 2022.

Builds on

58 of 138 references.

Closest ancestors · 14

Same direction, newest first

  1. 2021IQ-Learn: Inverse soft-Q Learning for Imitation23 Jun
  2. 2020Error Bounds of Imitating Policies and Environments22 Oct
  3. 2020Strictly Batch Imitation Learning by Energy-based Distribution Matching25 Jun
  4. 2020Adversarial Soft Advantage Fitting: Imitation Learning without Policy Optimization23 Jun
  5. 2020Primal Wasserstein Imitation Learning8 Jun
  6. 2020Safe Imitation Learning via Fast Bayesian Reward Inference from Preferences21 Feb
  7. 2020On Computation and Generalization of Generative Adversarial Imitation Learning9 Jan
  8. 2019Imitation Learning via Off-Policy Distribution Matching10 Dec
  9. 2019Wasserstein Adversarial Imitation Learning19 Jun
  10. 2019Imitation Learning as f-Divergence Minimization30 May
  11. 2019SQIL: Imitation Learning via Reinforcement Learning with Sparse Rewards27 May
  12. 2019On the Global Convergence of Imitation Learning: A Case for Linear Quadratic Regulator11 Jan
2 more
  1. 2018An Algorithmic Perspective on Imitation Learning16 Nov
  2. 2018Discriminator-Actor-Critic: Addressing Sample Inefficiency and Reward Bias in Adversarial Imitation Learning9 Sep

Other references · 44

  1. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  2. 2017Proximal Policy Optimization Algorithmscited by 14,441
  3. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  4. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  5. 2016OpenAI Gymcited by 1,684
  6. 2016Generative Adversarial Imitation Learningcited by 1,576
  7. 2019Introduction to Online Convex Optimizationcited by 793
  8. 2019A Theory of Regularized Markov Decision Processescited by 279
  9. 2020Model-Based Reinforcement Learning with Value-Targeted Regressioncited by 233
  10. 2020FLAMBE: Structural Complexity and Representation Learning of Low Rank MDPscited by 232
  11. 2018Time-uniform, nonparametric, nonasymptotic confidence sequencescited by 220
  12. 2019Provably Efficient Exploration in Policy Optimizationcited by 203
32 more
  1. 2017A unified view of entropy-regularized Markov decision processescited by 174
  2. 2020Minimax-Optimal Off-Policy Evaluation with Linear Function Approximationcited by 120
  3. 2021Reward (Mis)design for Autonomous Drivingcited by 102
  4. 2020On Reward-Free Reinforcement Learning with Linear Function Approximationcited by 89
  5. 2020Information Theoretic Regret Bounds for Online Nonlinear Controlcited by 87
  6. 2020Reinforcement Learning in Economics and Financecited by 57
  7. 2021Scalable Bayesian Inverse Reinforcement Learningcited by 56
  8. 2016Model-Free Imitation Learning with Policy Optimizationcited by 52
  9. 2020Efficiently Solving MDPs with Stochastic Mirror Descentcited by 52
  10. 2020A Unifying View of Optimism in Episodic Reinforcement Learningcited by 50
  11. 2019Variance Reduction for Matrix Gamescited by 37
  12. 2018A Forward-Backward Splitting Method for Monotone Inclusions Without Cocoercivitycited by 35
  13. 2020Logistic Q-Learningcited by 34
  14. 2020Online learning in MDPs with linear function approximation and bandit feedbackcited by 29
  15. 2021Online Apprenticeship Learningcited by 28
  16. 2020Online Sparse Reinforcement Learningcited by 25
  17. 2018Accelerated Bregman Proximal Gradient Methods for Relatively Smooth Convex Optimizationcited by 20
  18. 2020Efficient Planning in Large MDPs with Weak Linear Function Approximationcited by 17
  19. 2020Deep Inverse Q-learning with Constraintscited by 17
  20. 2014Linear Programming for Large-Scale Markov Decision Problemscited by 12
  21. 2018Scalable Bilinear π Learning Using State and Action Featurescited by 10
  22. 2019A Reduction from Reinforcement Learning to No-Regret Online Learningcited by 10
  23. 2017A Linearly Relaxed Approximate Linear Program for Markov Decision Processescited by 9
  24. 2020A maximum-entropy approach to off-policy evaluation in average-reward MDPscited by 9
  25. 2021Efficient Performance Bounds for Primal-Dual Reinforcement Learning from Demonstrationscited by 9
  26. 2019Exploration-Enhanced POLITEXcited by 8
  27. 2019Faster saddle-point optimization for solving large-scale Markov decision processescited by 8
  28. 2017From Infinite to Finite Programs: Explicit Error Bounds with Applications to Approximate Dynamic Programmingcited by 4
  29. 2020Convex Q-Learning, Part 1: Deterministic Optimal Controlcited by 4
  30. 2016Performance guarantees for model-based Approximate Dynamic Programming in continuous spacescited by 3
  31. 2020Bregman Augmented Lagrangian and Its Accelerationcited by 3
  32. 2021Near Optimal Policy Optimization via REPScited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 1; 2023: 3; 2024: 7; 2025: 9; 2026: 5.

25 papers.

Sort:

By area:

  1. 2023Dual RL: Unification and New Methods for Reinforcement and Imitation Learningcited by 48
  2. 2023Coherent Soft Imitation Learningcited by 21
  3. 2024Imitating Language via Scalable Inverse Reinforcement Learningcited by 21
  4. 2023Provably Efficient Adversarial Imitation Learning with Unknown Transitionscited by 16
  5. 2024Imitation Learning in Discounted Linear MDPs without exploration assumptionscited by 15
  6. 2022Efficient Global Planning in Large MDPs via Stochastic Primal-Dual Optimizationcited by 8
  7. 2024OLLIE: Imitation Learning from Offline Pretraining to Online Finetuningcited by 6
  8. 2024Provably and Practically Efficient Adversarial Imitation Learning with General Function Approximationcited by 6
  9. 2025IL-SOAR : Imitation Learning with Soft Optimistic Actor cRiticcited by 5
  10. 2025Inverse Q-Learning Done Right: Offline Imitation Learning in Qπ-Realizable MDPscited by 5
  11. 2025Learning Equilibria from Data: Provably Efficient Multi-Agent Imitation Learningcited by 4
  12. 2025Multi-Step Alignment as Markov Games: An Optimistic Online Gradient Descent Approach with Convergence Guaranteescited by 3
13 more
  1. 2024Expert Proximity as Surrogate Rewards for Single Demonstration Imitation Learningcited by 2
  2. 2024Convergence of a model-free entropy-regularized inverse reinforcement learning algorithmcited by 2
  3. 2025Beyond-Expert Performance with Limited Demonstrations: Efficient Imitation Learning with Double Explorationcited by 1
  4. 2025Trust Region Reward Optimization and Proximal Inverse Reward Optimization Algorithmcited by 1
  5. 2026Multi-agent imitation learning with function approximation: Linear Markov games and beyondcited by 1
  6. 2026Adversarial Imitation Learning with General Function Approximation: Theoretical Analysis and Practical Algorithmscited by 1
  7. 2024Randomized algorithms and PAC bounds for inverse reinforcement learning in continuous spacescited by 0
  8. 2025RIZE: Adaptive Regularization for Imitation Learningcited by 0
  9. 2025Value from Observations: Towards Large-Scale Imitation Learning via Self-Improvementcited by 0
  10. 2025A proximal augmented Lagrangian method for nonconvex optimization with equality and inequality constraintscited by 0
  11. 2026Non-Adversarial Imitation Learning Provably Free of Compounding Errors: The Value Flow Mechanismcited by 0
  12. 2026Split the Differences, Pool the Rest: Provably Efficient Multi-Objective Imitationcited by 0
  13. 2026Provably Efficient Policy-Reward Co-Pretraining for Adversarial Imitation Learningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Offline Imitation Learning by Controlling the Effective Planning Horizon
  2. 2022Improved Policy Optimization for Online Imitation Learning
  3. 2022Minimax Optimal Online Imitation Learning via Replay Estimation
  4. 2025The Pitfalls of Imitation Learning when Actions are Continuous
  5. 2023Offline Imitation Learning with Suboptimal Demonstrations via Relaxed Distribution Matching
  6. 2019Reinforced Imitation in Heterogeneous Action Space
  7. 2021Provably Breaking the Quadratic Error Compounding Barrier in Imitation Learning, Optimally
  8. 2024Is Behavior Cloning All You Need? Understanding Horizon in Imitation Learning