mlsanityData to 7 Oct 2026

A Simple Solution for Offline Imitation from Observations and Examples with Possibly Incomplete Trajectories

arXiv 2311.01329 ↗First posted 2 Nov 2023cs.LG cs.AIcited by 3

Not on the map. Why?

Abstract (arXiv)

Offline imitation from observations aims to solve MDPs where only task-specific expert states and task-agnostic non-expert state-action pairs are available. Offline imitation is useful in real-world scenarios where arbitrary interactions are costly and expert actions are unavailable. The state-of-the-art "DIstribution Correction Estimation" (DICE) methods minimize divergence of state occupancy between expert and learner policies and retrieve a policy with weighted behavior cloning; however, their results are unstable when learning from incomplete trajectories, due to a non-robust optimization in the dual domain. To address the issue, in this paper, we propose Trajectory-Aware Imitation Learning from Observations (TAILO). TAILO uses a discounted sum along the future trajectory as the weight for weighted behavior cloning. The terms for the sum are scaled by the output of a discriminator, which aims to identify expert states. Despite simplicity, TAILO works well if there exist trajectories or segments of expert behavior in the task-agnostic data, a common assumption in prior work. In experiments across multiple testbeds, we find TAILO to be more robust and effective, particularly with incomplete trajectories.

Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Builds on 46 of its 71 references that are in the map. First posted on arXiv 2 Nov 2023.

Builds on

46 of 71 references.

References · 46

  1. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,302
  2. 2016Generative Adversarial Imitation Learningcited by 1,576
  3. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learningcited by 1,406
  4. 2019Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learningcited by 750
  5. 2017Mastering Chess and Shogi by Self-Play with a General Reinforcement Learning Algorithmcited by 723
  6. 2018MINE: Mutual Information Neural Estimationcited by 680
  7. 2020MOReL : Model-Based Offline Reinforcement Learningcited by 582
  8. 2017Learning Robust Rewards with Adversarial Inverse Reinforcement Learningcited by 485
  9. 2018Behavioral Cloning from Observationcited by 482
  10. 2020Is Pessimism Provably Efficient for Offline RL?cited by 362
  11. 2018Maximum a Posteriori Policy Optimisationcited by 318
  12. 2020Critic Regularized Regressioncited by 269
34 more
  1. 2019DualDICE: Behavior-Agnostic Estimation of Discounted Stationary Distribution Correctionscited by 254
  2. 2022Is Reinforcement Learning (Not) for Natural Language Processing: Benchmarks, Baselines, and Building Blocks for Natural Language Policy Optimizationcited by 244
  3. 2020Accelerating Reinforcement Learning with Learned Skill Priorscited by 220
  4. 2017Positive-Unlabeled Learning with Non-Negative Risk Estimatorcited by 217
  5. 2018Discriminator-Actor-Critic: Addressing Sample Inefficiency and Reward Bias in Adversarial Imitation Learningcited by 203
  6. 2019A Divergence Minimization Perspective on Imitation Learning Methodscited by 188
  7. 2018Generative Adversarial Imitation from Observationcited by 183
  8. 2019Imitation Learning via Off-Policy Distribution Matchingcited by 182
  9. 2021The Surprising Effectiveness of Representation Learning for Visual Imitationcited by 149
  10. 2021OptiDICE: Offline Policy Optimization via Stationary Distribution Correction Estimationcited by 121
  11. 2021Learning Generalizable Robotic Reward Functions from "In-The-Wild" Human Videoscited by 116
  12. 2019Imitation Learning from Imperfect Demonstrationcited by 112
  13. 2020Toward the Fundamental Limits of Imitation Learningcited by 96
  14. 2022Discriminator-Weighted Offline Imitation Learning from Suboptimal Demonstrationscited by 89
  15. 2021Demonstration-Guided Reinforcement Learning with Learned Skillscited by 73
  16. 2020Offline Learning from Demonstrations and Unlabeled Experiencecited by 72
  17. 2021Off-Policy Imitation Learning from Observationscited by 70
  18. 2018Generative Adversarial User Model for Reinforcement Learning Based Recommendation Systemcited by 64
  19. 2020ReLMoGen: Leveraging Motion Generation in Reinforcement Learning for Mobile Manipulationcited by 47
  20. 2023Optimal Transport for Offline Imitation Learningcited by 42
  21. 2021Replacing Rewards with Examples: Example-Based Policy Search via Recursive Classificationcited by 38
  22. 2021Hierarchical Few-Shot Imitation with Skill Transition Modelscited by 37
  23. 2019Positive-Unlabeled Reward Learningcited by 36
  24. 2021MURAL: Meta-Learning Uncertainty-Aware Rewards for Outcome-Driven Reinforcement Learningcited by 24
  25. 2022LobsDICE: Offline Learning from Observation via Stationary Distribution Correction Estimationcited by 22
  26. 2023MAHALO: Unifying Offline Reinforcement Learning and Imitation Learning from Observationscited by 18
  27. 2022Rethinking ValueDice: Does It Really Improve Performance?cited by 15
  28. 2023A Coupled Flow Approach to Imitation Learningcited by 12
  29. 2019Model-based Behavioral Cloning with Future Image Similarity Learningcited by 8
  30. 2020Off-Policy Adversarial Inverse Reinforcement Learningcited by 8
  31. 2021On Generalization of Adversarial Imitation Learning and Beyondcited by 7
  32. 2021Reward-Weighted Regression Converges to a Global Optimumcited by 5
  33. 2017Hierarchical Policy Search via Return-Weighted Density Estimationcited by 4
  34. 2019PUNCH: Positive UNlabelled Classification based information retrieval in Hyperspectral imagescited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 2; 2025: 0; 2026: 1.

3 papers.

  1. 2024In-Trajectory Inverse Reinforcement Learning: Learn Incrementally Before An Ongoing Trajectory Terminatescited by 7
  2. 2024Offline Reinforcement Learning with Domain-Unlabeled Datacited by 3
  3. 2026Latent Wasserstein Adversarial Imitation Learningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2021Mitigating Covariate Shift in Imitation Learning via Offline Data Without Great Coverage
  2. 2024Offline Imitation Learning by Controlling the Effective Planning Horizon
  3. 2025Robust Offline Imitation Learning Through State-level Trajectory Stitching
  4. 2024A Dual Approach to Imitation Learning from Observations with Offline Datasets
  5. 2023GO-DICE: Goal-Conditioned Option-Aware Offline Imitation Learning via Stationary Distribution Correction Estimation
  6. 2025Learning What to Do and What Not To Do: Offline Imitation from Expert and Undesirable Demonstrations
  7. 2024How to Leverage Diverse Demonstrations in Offline Imitation Learning
  8. 2025Learning from Observation: A Survey of Recent Advances