mlsanityData to 7 Oct 2026

Versatile Offline Imitation from Observations and Examples via Regularized State-Occupancy Matching

arXiv 2202.02433 ↗First posted 4 Feb 2022cs.LG cs.AIcited by 43

On the map: Offline RL and imitation learning from data → Imitation learning methods

Abstract (arXiv)

We propose State Matching Offline DIstribution Correction Estimation (SMODICE), a novel and versatile regression-based offline imitation learning (IL) algorithm derived via state-occupancy matching. We show that the SMODICE objective admits a simple optimization procedure through an application of Fenchel duality and an analytic solution in tabular MDPs. Without requiring access to expert actions, SMODICE can be effectively applied to three offline IL settings: (i) imitation from observations (IfO), (ii) IfO with dynamics or morphologically mismatched expert, and (iii) example-based reinforcement learning, which we show can be formulated as a state-occupancy matching problem. We extensively evaluate SMODICE on both gridworld environments as well as on high-dimensional offline benchmarks. Our results demonstrate that SMODICE is effective for all three problem settings and significantly outperforms prior state-of-art.

Key facts. Cited by 43 papers in the map (data through 7 Oct 2026). Placed in Imitation learning methods, one of 13 directions in Offline RL and imitation learning from data. Builds on 35 of its 48 references that are in the map. First posted on arXiv 4 Feb 2022.

Builds on

35 of 48 references.

Closest ancestors · 13

Same direction, newest first

  1. 2021Mitigating Covariate Shift in Imitation Learning via Offline Data Without Great Coverage6 Jun
  2. 2021Cross-domain Imitation from Observations20 May
  3. 2021Off-Policy Imitation Learning from Observations25 Feb
  4. 2020Offline Learning from Demonstrations and Unlabeled Experience27 Nov
  5. 2020State-only Imitation with Transition Dynamics Mismatch27 Feb
  6. 2019Imitation Learning via Off-Policy Distribution Matching10 Dec
  7. 2019State Alignment-based Imitation Learning21 Nov
  8. 2019A Divergence Minimization Perspective on Imitation Learning Methods6 Nov
  9. 2019Imitation Learning from Observations by Minimizing Inverse Dynamics Disagreement10 Oct
  10. 2019Imitation Learning as f-Divergence Minimization30 May
  11. 2018Discriminator-Actor-Critic: Addressing Sample Inefficiency and Reward Bias in Adversarial Imitation Learning9 Sep
  12. 2018Generative Adversarial Imitation from Observation17 Jul
1 more
  1. 2018Behavioral Cloning from Observation4 May

Other references · 22

  1. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  2. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  3. 2021Convex optimizationcited by 2,479
  4. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problemscited by 1,873
  5. 2020Conservative Q-Learning for Offline Reinforcement Learningcited by 1,777
  6. 2016Generative Adversarial Imitation Learningcited by 1,576
  7. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learningcited by 1,406
  8. 2021A Minimalist Approach to Offline Reinforcement Learningcited by 882
  9. 2019Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reductioncited by 833
  10. 2020MOPO: Model-based Offline Policy Optimizationcited by 630
  11. 2020MOReL : Model-Based Offline Reinforcement Learningcited by 582
  12. 2019Relay Policy Learning: Solving Long-Horizon Tasks via Imitation and Reinforcement Learningcited by 380
10 more
  1. 2019DualDICE: Behavior-Agnostic Estimation of Discounted Stationary Distribution Correctionscited by 254
  2. 2019AlgaeDICE: Policy Gradient from Arbitrary Experiencecited by 187
  3. 2020GenDICE: Generalized Offline Estimation of Stationary Valuescited by 122
  4. 2021OptiDICE: Offline Policy Optimization via Stationary Distribution Correction Estimationcited by 121
  5. 2020Reinforcement Learning via Fenchel-Rockafellar Dualitycited by 100
  6. 2020State-Only Imitation Learning for Dexterous Manipulationcited by 90
  7. 2021Conservative Offline Distributional Reinforcement Learningcited by 78
  8. 2020CoinDICE: Off-Policy Confidence Interval Estimationcited by 64
  9. 2021Replacing Rewards with Examples: Example-Based Policy Search via Recursive Classificationcited by 38
  10. 2021Conservative and Adaptive Penalty for Model-Based Safe Reinforcement Learningcited by 13

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 9; 2024: 19; 2025: 11; 2026: 4.

43 papers.

Sort:

By area:

  1. 2024Correcting the Mythos of KL-Regularization: Direct Alignment without Overoptimization via Chi-Squared Preference Optimizationcited by 78
  2. 2023Beyond Uniform Sampling: Offline Reinforcement Learning with Imbalanced Datasetscited by 24
  3. 2023Harnessing Mixed Offline Reinforcement Learning Datasets via Trajectory Weightingcited by 22
  4. 2023Coherent Soft Imitation Learningcited by 21
  5. 2024Imitating Language via Scalable Inverse Reinforcement Learningcited by 21
  6. 2023Survival Instinct in Offline Reinforcement Learningcited by 20
  7. 2024Stitching Sub-Trajectories with Conditional Diffusion Model for Goal-Conditioned Offline RLcited by 20
  8. 2023MAHALO: Unifying Offline Reinforcement Learning and Imitation Learning from Observationscited by 18
  9. 2023Mind the Gap: Offline Policy Optimization for Imperfect Rewardscited by 15
  10. 2024SEABO: A Simple Search-Based Method for Offline Imitation Learningcited by 14
  11. 2024Adaptive Advantage-Guided Policy Regularization for Offline Reinforcement Learningcited by 13
  12. 2026Large Reward Models: Generalizable Online Robot Reward Generation with Vision-Language Modelscited by 11
31 more
  1. 2024Align Your Intents: Offline Imitation Learning via Optimal Transportcited by 7
  2. 2024Constrained Ensemble Exploration for Unsupervised Skill Discoverycited by 7
  3. 2023GO-DICE: Goal-Conditioned Option-Aware Offline Imitation Learning via Stationary Distribution Correction Estimationcited by 6
  4. 2024OLLIE: Imitation Learning from Offline Pretraining to Online Finetuningcited by 6
  5. 2024Hybrid Reinforcement Learning from Offline Observation Alonecited by 5
  6. 2025IL-SOAR : Imitation Learning with Soft Optimistic Actor cRiticcited by 5
  7. 2023Offline Diversity Maximization Under Imitation Constraintscited by 4
  8. 2024OMPO: A Unified Framework for RL under Policy and Dynamics Shiftscited by 4
  9. 2024Bellman Diffusion Modelscited by 4
  10. 2023Imitation Learning from Purified Demonstrationscited by 3
  11. 2024Context-Former: Stitching via Latent Conditioned Sequence Modelingcited by 3
  12. 2024Markov Balance Satisfaction Improves Performance in Strictly Batch Offline Imitation Learningcited by 3
  13. 2024Zero-Shot Offline Imitation Learning via Optimal Transportcited by 3
  14. 2025Offline Imitation Learning upon Arbitrary Demonstrations by Pre-Training Dynamics Representationscited by 3
  15. 2025Dual-Force: Enhanced Offline Diversity Maximization under Imitation Constraintscited by 2
  16. 2025PROF: An LLM-based Reward Code Preference Optimization Framework for Offline Imitation Learningcited by 2
  17. 2024Provably Efficient Off-Policy Adversarial Imitation Learning with Convergence Guaranteescited by 1
  18. 2025SEMDICE: Off-policy State Entropy Maximization via Stationary Distribution Correction Estimationcited by 1
  19. 2024Offline Imitation Learning by Controlling the Effective Planning Horizoncited by 0
  20. 2024Imitating from auxiliary imperfect demonstrations via Adversarial Density Weighted Regressioncited by 0
  21. 2024Robust off-policy Reinforcement Learning via Soft Constrained Adversarycited by 0
  22. 2024UNIQ: Offline Inverse Q-learning for Avoiding Undesirable Demonstrationscited by 0
  23. 2025Robust Offline Imitation Learning Through State-level Trajectory Stitchingcited by 0
  24. 2025Learning What to Do and What Not To Do: Offline Imitation from Expert and Undesirable Demonstrationscited by 0
  25. 2025Minimal Ingredients for Reward Assignment from Expert Demonstrationscited by 0
  26. 2025Off-Policy Actor-Critic for Adversarial Observation Robustness: Virtual Alternative Training via Symmetric Policy Evaluationcited by 0
  27. 2025Value from Observations: Towards Large-Scale Imitation Learning via Self-Improvementcited by 0
  28. 2025Learning from Observation: A Survey of Recent Advancescited by 0
  29. 2026Imitation from Observations with Trajectory-Level Generative Embeddingscited by 0
  30. 2026Latent Wasserstein Adversarial Imitation Learningcited by 0
  31. 2026Long-Horizon Language Model Reinforcement Learning via Progressive Point Matchingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023A Simple Solution for Offline Imitation from Observations and Examples with Possibly Incomplete Trajectories
  2. 2023Offline Imitation Learning with Suboptimal Demonstrations via Relaxed Distribution Matching
  3. 2021Bridging Offline Reinforcement Learning and Imitation Learning: A Tale of Pessimism
  4. 2026How to Mitigate the Distribution Shift Problem in Robotics Control: A Robust and Adaptive Approach Based on Offline to Online Imitation Learning
  5. 2022A Policy-Guided Imitation Approach for Offline Reinforcement Learning
  6. 2023Using Offline Data to Speed Up Reinforcement Learning in Procedurally Generated Environments
  7. 2021SoftDICE for Imitation Learning: Rethinking Off-policy Distribution Matching
  8. 2025Using Non-Expert Data to Robustify Imitation Learning via Offline Reinforcement Learning