mlsanityData to 7 Oct 2026

Online Apprenticeship Learning

arXiv 2102.06924 ↗First posted 13 Feb 2021cs.LG stat.MLcited by 28

Not on the map. Why?

Cited by 22 in Offline RL and imitation learning from data, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

In Apprenticeship Learning (AL), we are given a Markov Decision Process (MDP) without access to the cost function. Instead, we observe trajectories sampled by an expert that acts according to some policy. The goal is to find a policy that matches the expert's performance on some predefined set of cost functions. We introduce an online variant of AL (Online Apprenticeship Learning; OAL), where the agent is expected to perform comparably to the expert while interacting with the environment. We show that the OAL problem can be effectively solved by combining two mirror descent based no-regret algorithms: one for policy optimization and another for learning the worst case cost. By employing optimistic exploration, we derive a convergent algorithm with O(√K) regret, where K is the number of interactions with the MDP, and an additional linear error term that depends on the amount of expert trajectories available. Importantly, our algorithm avoids the need to solve an MDP at each iteration, making it more practical compared to prior AL methods. Finally, we implement a deep variant of our algorithm which shares some similarities to GAIL citeho2016generative, but where the discriminator is replaced with the costs learned by the OAL problem. Our simulations suggest that OAL performs well in high dimensional control problems.

Key facts. Cited by 28 papers in the map (data through 7 Oct 2026). Builds on 26 of its 53 references that are in the map. First posted on arXiv 13 Feb 2021.

Builds on

26 of 53 references.

References · 26

  1. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  2. 2015Trust Region Policy Optimizationcited by 2,503
  3. 2017Improved Training of Wasserstein GANscited by 2,223
  4. 2016Generative Adversarial Imitation Learningcited by 1,576
  5. 2019Introduction to Online Convex Optimizationcited by 793
  6. 2016Deep Exploration via Bootstrapped DQNcited by 518
  7. 2019Online Learning: A Modern Introduction Using Convex Optimizationcited by 407
  8. 2019A Theory of Regularized Markov Decision Processescited by 279
  9. 2018Discriminator-Actor-Critic: Addressing Sample Inefficiency and Reward Bias in Adversarial Imitation Learningcited by 203
  10. 2019Provably Efficient Exploration in Policy Optimizationcited by 203
  11. 2019Tighter Problem-Dependent Regret Bounds in Reinforcement Learning without Domain Knowledge using Value Function Boundscited by 194
  12. 2017Unifying PAC and Regret: Uniform PAC Bounds for Episodic Reinforcement Learningcited by 175
14 more
  1. 2019On the Theory of Policy Gradient Methods: Optimality, Approximation, and Distribution Shiftcited by 174
  2. 2021Provably Efficient Reinforcement Learning with Linear Function Approximation Under Adaptivity Constraintscited by 124
  3. 2019Adaptive Trust Region Policy Optimization: Global Convergence and Faster Rates for Regularized MDPscited by 113
  4. 2020Mirror Descent Policy Optimizationcited by 102
  5. 2020Toward the Fundamental Limits of Imitation Learningcited by 96
  6. 2020Optimistic Policy Optimization with Bandit Feedbackcited by 79
  7. 2019Wasserstein Adversarial Imitation Learningcited by 59
  8. 2016Model-Free Imitation Learning with Policy Optimizationcited by 52
  9. 2019Tight Regret Bounds for Model-Based Reinforcement Learning with Greedy Policiescited by 38
  10. 2018Sample-Efficient Imitation Learning via Generative Adversarial Netscited by 36
  11. 2020On Computation and Generalization of Generative Adversarial Imitation Learningcited by 27
  12. 2019Apprenticeship Learning via Frank-Wolfecited by 13
  13. 2020Wasserstein Distance guided Adversarial Imitation Learning with Reward Shape Explorationcited by 13
  14. 2020Generative Adversarial Imitation Learning with Neural Networks: Global Optimality and Convergence Ratecited by 7

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 2; 2022: 4; 2023: 4; 2024: 5; 2025: 7; 2026: 6.

28 papers.

Sort:

By area:

  1. 2021Reward is enough for convex MDPscited by 82
  2. 2022Discovering Policies with DOMiNO: Diversity Optimization Maintaining Near Optimalitycited by 35
  3. 2022Active Exploration for Inverse Reinforcement Learningcited by 27
  4. 2023When Demonstrations Meet Generative World Models: A Maximum Likelihood Framework for Offline Inverse Reinforcement Learningcited by 26
  5. 2022Proximal Point Imitation Learningcited by 25
  6. 2023ReLOAD: Reinforcement Learning with Optimistic Ascent-Descent for Last-Iterate Convergence in Constrained MDPscited by 19
  7. 2023Provably Efficient Adversarial Imitation Learning with Unknown Transitionscited by 16
  8. 2023Diversifying AI: Towards Creative Chess with AlphaZerocited by 15
  9. 2024Imitation Learning in Discounted Linear MDPs without exploration assumptionscited by 15
  10. 2025Optimistically Optimistic Exploration for Provably Efficient Infinite-Horizon Reinforcement and Imitation Learningcited by 12
  11. 2021Provably Efficient Generative Adversarial Imitation Learning for Online and Offline Setting with Linear Function Approximationcited by 11
  12. 2022Improved Policy Optimization for Online Imitation Learningcited by 6
16 more
  1. 2024Provably and Practically Efficient Adversarial Imitation Learning with General Function Approximationcited by 6
  2. 2024How does Inverse RL Scale to Large State Spaces? A Provably Efficient Approachcited by 5
  3. 2025IL-SOAR : Imitation Learning with Soft Optimistic Actor cRiticcited by 5
  4. 2025Inverse Q-Learning Done Right: Offline Imitation Learning in Qπ-Realizable MDPscited by 5
  5. 2025Efficient Imitation under Misspecificationcited by 4
  6. 2024Convergence of a model-free entropy-regularized inverse reinforcement learning algorithmcited by 2
  7. 2025Near-Optimal Second-Order Guarantees for Model-Based Adversarial Imitation Learningcited by 2
  8. 2024Provably Efficient Off-Policy Adversarial Imitation Learning with Convergence Guaranteescited by 1
  9. 2025Beyond-Expert Performance with Limited Demonstrations: Efficient Imitation Learning with Double Explorationcited by 1
  10. 2026Adversarial Imitation Learning with General Function Approximation: Theoretical Analysis and Practical Algorithmscited by 1
  11. 2026Fast Rates for Inverse Reinforcement Learningcited by 1
  12. 2025Robustness in the Face of Partial Identifiability in Reward Learningcited by 0
  13. 2026Provably avoiding over-optimization in Direct Preference Optimization without knowing the data distributioncited by 0
  14. 2026Split the Differences, Pool the Rest: Provably Efficient Multi-Objective Imitationcited by 0
  15. 2026Provably Efficient Policy-Reward Co-Pretraining for Adversarial Imitation Learningcited by 0
  16. 2026Provable Benefits of Regularization: Fast Rates for Adversarial Imitation Learningcited by 0