mlsanityData to 14 Oct 2026

IL-SOAR : Imitation Learning with Soft Optimistic Actor cRitic

arXiv 2502.19859 ↗First posted 27 Feb 2025cs.LGcited by 5

On the map: Offline RL and imitation learning from data → Imitation learning methods

also in 1 more
Abstract (arXiv)

This paper introduces the SOAR framework for imitation learning. SOAR is an algorithmic template that learns a policy from expert demonstrations with a primal dual style algorithm that alternates cost and policy updates. Within the policy updates, the SOAR framework uses an actor critic method with multiple critics to estimate the critic uncertainty and build an optimistic critic fundamental to drive exploration. When instantiated in the tabular setting, we get a provable algorithm with guarantees that matches the best known results in ε. Practically, the SOAR template is shown to boost consistently the performance of imitation learning algorithms based on Soft Actor Critic such as f-IRL, ML-IRL and CSIL in several MuJoCo environments. Overall, thanks to SOAR, the required number of episodes to achieve the same performance is reduced by half.

Key facts. Cited by 5 papers in the map (data through 14 Oct 2026). Placed in Imitation learning methods, one of 13 directions in Offline RL and imitation learning from data. Builds on 94 of its 117 references that are in the map. First posted on arXiv 27 Feb 2025.

Builds on

94 of 117 references.

Closest ancestors · 38

Same direction, newest first

  1. 2024Provably and Practically Efficient Adversarial Imitation Learning with General Function Approximation1 Nov
  2. 2024Diffusing States and Matching Scores: A New Framework for Imitation Learning17 Oct
  3. 2024Is Behavior Cloning All You Need? Understanding Horizon in Imitation Learning20 Jul
  4. 2024A Dual Approach to Imitation Learning from Observations with Offline Datasets13 Jun
  5. 2024Imitation Learning in Discounted Linear MDPs without exploration assumptions3 May
  6. 2024Adversarial Imitation Learning via Boosting12 Apr
  7. 2023Imitation Learning from Observation through Optimal Transport2 Oct
  8. 2023Learning non-Markovian Decision-Making from State-only Sequences27 Jun
  9. 2023Provably Efficient Adversarial Imitation Learning with Unknown Transitions11 Jun
  10. 2023Coherent Soft Imitation Learning25 May
  11. 2023Offline Imitation Learning with Suboptimal Demonstrations via Relaxed Distribution Matching5 Mar
  12. 2022Proximal Point Imitation Learning22 Sep
26 more
  1. 2022Minimax Optimal Online Imitation Learning via Replay Estimation30 May
  2. 2022Imitation Learning from Observations under Transition Model Disparity25 Apr
  3. 2022Robust Learning from Observation with Model Misspecification12 Feb
  4. 2022A Ranking Game for Imitation Learning7 Feb
  5. 2022Versatile Offline Imitation from Observations and Examples via Regularized State-Occupancy Matching4 Feb
  6. 2021IQ-Learn: Inverse soft-Q Learning for Imitation23 Jun
  7. 2021What Matters for Adversarial Imitation Learning?1 Jun
  8. 2021Learning from Imperfect Demonstrations from Agents with Varying Dynamics10 Mar
  9. 2021Of Moments and Matching: A Game-Theoretic Framework for Closing the Imitation Gap4 Mar
  10. 2021Off-Policy Imitation Learning from Observations25 Feb
  11. 2021Provably Breaking the Quadratic Error Compounding Barrier in Imitation Learning, Optimally25 Feb
  12. 2021MobILE: Model-Based Imitation Learning From Observation Alone22 Feb
  13. 2020Toward the Fundamental Limits of Imitation Learning13 Sep
  14. 2020Intrinsic Reward Driven Imitation Learning via Generative Model26 Jun
  15. 2020Primal Wasserstein Imitation Learning8 Jun
  16. 2020State-only Imitation with Transition Dynamics Mismatch27 Feb
  17. 2020Provable Representation Learning for Imitation Learning via Bi-level Optimization24 Feb
  18. 2019State Alignment-based Imitation Learning21 Nov
  19. 2019Imitation Learning from Observations by Minimizing Inverse Dynamics Disagreement10 Oct
  20. 2019Recent Advances in Imitation Learning from Observation30 May
  21. 2019SQIL: Imitation Learning via Reinforcement Learning with Sparse Rewards27 May
  22. 2019Provably Efficient Imitation Learning from Observation Alone27 May
  23. 2018Discriminator-Actor-Critic: Addressing Sample Inefficiency and Reward Bias in Adversarial Imitation Learning9 Sep
  24. 2018Generative Adversarial Imitation from Observation17 Jul
  25. 2018Imitating Latent Policies from Observation21 May
  26. 2018Behavioral Cloning from Observation4 May

Other references · 56

  1. 2017Proximal Policy Optimization Algorithmscited by 15,215
  2. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
  3. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,417
  4. 2015Deep Reinforcement Learning with Double Q-learningcited by 1,745
  5. 2016Generative Adversarial Imitation Learningcited by 1,597
  6. 2017Curiosity-driven Exploration by Self-supervised Predictioncited by 1,194
  7. 2018Deep Reinforcement Learning in a Handful of Trials using Probabilistic Dynamics Modelscited by 774
  8. 2016Deep Exploration via Bootstrapped DQNcited by 527
  9. 2017Time-Contrastive Networks: Self-Supervised Learning from Videocited by 459
  10. 2018Large-Scale Study of Curiosity-Driven Learningcited by 280
  11. 2018Model-Ensemble Trust-Region Policy Optimizationcited by 219
  12. 2018Randomized Prior Functions for Deep Reinforcement Learningcited by 219
38 more
  1. 2021Epistemic Neural Networkscited by 120
  2. 2020Effective Diversity in Population Based Reinforcement Learningcited by 119
  3. 2018Zero-Shot Visual Imitationcited by 117
  4. 2014Generalization and Exploration via Randomized Value Functionscited by 112
  5. 2016Averaged-DQN: Variance Reduction and Stabilization for Deep Reinforcement Learningcited by 106
  6. 2020State-Only Imitation Learning for Dexterous Manipulationcited by 91
  7. 2017Combining Self-Supervised Learning and Imitation for Vision-Based Rope Manipulationcited by 90
  8. 2019Better Exploration with Optimistic Actor-Criticcited by 88
  9. 2020Efficient Model-Based Reinforcement Learning through Optimistic Policy Search and Planningcited by 86
  10. 2017Learning Invariant Feature Spaces to Transfer Skills with Reinforcement Learningcited by 85
  11. 2017The Uncertainty Bellman Equation and Explorationcited by 82
  12. 2019Learning Linear-Quadratic Regulators Efficiently with only √T Regretcited by 70
  13. 2017Boltzmann Exploration Done Rightcited by 69
  14. 2020Munchausen Reinforcement Learningcited by 69
  15. 2020f-IRL: Inverse Reinforcement Learning via State Marginal Matchingcited by 67
  16. 2021Tactical Optimism and Pessimism for Deep Reinforcement Learningcited by 55
  17. 2016Model-Free Imitation Learning with Policy Optimizationcited by 52
  18. 2022Exploration via Elliptical Episodic Bonusescited by 51
  19. 2021Randomized Exploration for Reinforcement Learning with General Value Function Approximationcited by 49
  20. 2022Maximum-Likelihood Inverse Reinforcement Learning with Finite-Time Guaranteescited by 42
  21. 2014Near-optimal Reinforcement Learning in Factored MDPscited by 41
  22. 2023Inverse Reinforcement Learning without Reinforcement Learningcited by 36
  23. 2020Robust Inverse Reinforcement Learning under Transition Dynamics Mismatchcited by 33
  24. 2024Hybrid Inverse Reinforcement Learningcited by 32
  25. 2018Variational Bayesian Reinforcement Learning with Regret Boundscited by 31
  26. 2023Approximate Thompson Sampling via Epistemic Neural Networkscited by 31
  27. 2021Online Apprenticeship Learningcited by 29
  28. 2023Provable and Practical: Efficient Exploration in Reinforcement Learning via Langevin Monte Carlocited by 29
  29. 2022LobsDICE: Offline Learning from Observation via Stationary Distribution Correction Estimationcited by 23
  30. 2018Dual Policy Iterationcited by 22
  31. 2021Efficient Continuous Control with Double Actors and Regularized Criticscited by 21
  32. 2024Imitating Language via Scalable Inverse Reinforcement Learningcited by 21
  33. 2025Langevin Soft Actor-Critic: Efficient Exploration through Uncertainty-Driven Critic Learningcited by 17
  34. 2023Model-Based Uncertainty in Value Functionscited by 15
  35. 2024More Efficient Randomized Exploration for Reinforcement Learning via Approximate Samplingcited by 12
  36. 2025Optimistically Optimistic Exploration for Provably Efficient Infinite-Horizon Reinforcement and Imitation Learningcited by 12
  37. 2019Deep Model-Based Reinforcement Learning via Estimated Uncertainty and Conservative Policy Optimizationcited by 11
  38. 2023Probabilistic Inference in Reinforcement Learning Done Rightcited by 11

50 most cited of 56 All 56 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 1; 2026: 4.

5 papers.

  1. 2025Learning Equilibria from Data: Provably Efficient Multi-Agent Imitation Learningcited by 4
  2. 2026Multi-agent imitation learning with function approximation: Linear Markov games and beyondcited by 1
  3. 2026Provably avoiding over-optimization in Direct Preference Optimization without knowing the data distributioncited by 0
  4. 2026Non-Adversarial Imitation Learning Provably Free of Compounding Errors: The Value Flow Mechanismcited by 0
  5. 2026Quantile of Means: A Bonus-Free Ensemble Method for Minimax Optimal Reinforcement Learningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Beyond-Expert Performance with Limited Demonstrations: Efficient Imitation Learning with Double Exploration
  2. 2022Regularized Soft Actor-Critic for Behavior Transfer Learning
  3. 2018Fast Policy Learning through Imitation and Reinforcement
  4. 2022Improved Policy Optimization for Online Imitation Learning
  5. 2025Unlocking the Potential of Soft Actor-Critic for Imitation Learning
  6. 2019Reinforced Imitation in Heterogeneous Action Space
  7. 2024Interactive and Hybrid Imitation Learning: Provably Beating Behavior Cloning
  8. 2018Self-Imitation Learning