# Sequence Model Imitation Learning with Unobserved Contexts arXiv: https://arxiv.org/abs/2208.02225 ยท HTML: https://mlsanity.com/paper/2208.02225/ ## Key facts - Cited by 34 papers in the map (data through 14 Oct 2026). - Placed in Imitation learning methods, one of 13 directions in Offline RL and imitation learning from data. - Builds on 22 of its 40 references that are in the map. - First posted on arXiv 3 Aug 2022. ## Placed in - Offline RL and imitation learning from data / [Imitation learning methods](https://mlsanity.com/offlinerl/imitation-learning-methods/) ## Builds on (most cited first, 22 of 22) - [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014) - [Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks](https://mlsanity.com/paper/1703.03400/) (2017) - [Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor](https://mlsanity.com/paper/1801.01290/) (2018) - [Generative Adversarial Imitation Learning](https://mlsanity.com/paper/1606.03476/) (2016) - [Learning Quadrupedal Locomotion over Challenging Terrain](https://mlsanity.com/paper/2010.11251/) (2020) - [RMA: Rapid Motor Adaptation for Legged Robots](https://mlsanity.com/paper/2107.04034/) (2021) - [ChauffeurNet: Learning to Drive by Imitating the Best and Synthesizing the Worst](https://mlsanity.com/paper/1812.03079/) (2018) - [Learning by Cheating](https://mlsanity.com/paper/1912.12294/) (2019) - [Exploring the Limitations of Behavior Cloning for Autonomous Driving](https://mlsanity.com/paper/1904.08980/) (2019) - [Causal Confusion in Imitation Learning](https://mlsanity.com/paper/1905.11979/) (2019) - [Contextual Markov Decision Processes](https://mlsanity.com/paper/1502.02259/) (2015) - [Imitating Driver Behavior with Generative Adversarial Networks](https://mlsanity.com/paper/1701.06699/) (2017) - [Of Moments and Matching: A Game-Theoretic Framework for Closing the Imitation Gap](https://mlsanity.com/paper/2103.03236/) (2021) - [Fighting Copycat Agents in Behavioral Cloning from Observation Histories](https://mlsanity.com/paper/2010.14876/) (2020) - [Feedback in Imitation Learning: The Three Regimes of Covariate Shift](https://mlsanity.com/paper/2102.02872/) (2021) - [Shaking the foundations: delusions in sequence models for interaction and control](https://mlsanity.com/paper/2110.10819/) (2021) - [Data-driven Planning via Imitation Learning](https://mlsanity.com/paper/1711.06391/) (2017) - [Causal Imitation Learning under Temporally Correlated Noise](https://mlsanity.com/paper/2202.01312/) (2022) - [Keyframe-Focused Visual Imitation Learning](https://mlsanity.com/paper/2106.06452/) (2021) - [Fighting Fire with Fire: Avoiding DNN Shortcuts through Priming](https://mlsanity.com/paper/2206.10816/) (2022) - [On Covariate Shift of Latent Confounders in Imitation and Reinforcement Learning](https://mlsanity.com/paper/2110.06539/) (2021) - [Resolving Copycat Problems in Visual Imitation Learning via Residual Action Prediction](https://mlsanity.com/paper/2207.09705/) (2022) ## Built on by (most cited first, 30 of 34) - [Self-Distillation Enables Continual Learning](https://mlsanity.com/paper/2601.19897/) (2026) - [A Minimaximalist Approach to Reinforcement Learning from Human Feedback](https://mlsanity.com/paper/2401.04056/) (2024) - [Data Quality in Imitation Learning](https://mlsanity.com/paper/2306.02437/) (2023) - [Better than Your Teacher: LLM Agents that learn from Privileged AI Feedback](https://mlsanity.com/paper/2410.05434/) (2024) - [Privileged Sensing Scaffolds Reinforcement Learning](https://mlsanity.com/paper/2405.14853/) (2024) - [BPP: Long-Context Robot Imitation Learning by Focusing on Key History Frames](https://mlsanity.com/paper/2602.15010/) (2026) - [TGRL: An Algorithm for Teacher Guided Reinforcement Learning](https://mlsanity.com/paper/2307.03186/) (2023) - [A Smooth Sea Never Made a Skilled SAILOR: Robust Imitation via Learning to Search](https://mlsanity.com/paper/2506.05294/) (2025) - [Structured Agent Distillation for Large Language Model](https://mlsanity.com/paper/2505.13820/) (2025) - [Multi-Agent Imitation Learning: Value is Easy, Regret is Hard](https://mlsanity.com/paper/2406.04219/) (2024) - [Delphic Offline Reinforcement Learning under Nonidentifiable Hidden Confounding](https://mlsanity.com/paper/2306.01157/) (2023) - [Student-Informed Teacher Training](https://mlsanity.com/paper/2412.09149/) (2024) - [Deconfounding Imitation Learning with Variational Inference](https://mlsanity.com/paper/2211.02667/) (2022) - [Efficient Imitation under Misspecification](https://mlsanity.com/paper/2503.13162/) (2025) - [Efficient Reinforcement Learning with Impaired Observability: Learning to Act with Delayed and Missing State Observations](https://mlsanity.com/paper/2306.01243/) (2023) - [Adversarial Moment-Matching Distillation of Large Language Models](https://mlsanity.com/paper/2406.02959/) (2024) - [IGDrivSim: A Benchmark for the Imitation Gap in Autonomous Driving](https://mlsanity.com/paper/2411.04653/) (2024) - [Your Learned Constraint is Secretly a Backward Reachable Tube](https://mlsanity.com/paper/2501.15618/) (2025) - [To Distill or Decide? Understanding the Algorithmic Trade-off in Partially Observable Reinforcement Learning](https://mlsanity.com/paper/2510.03207/) (2025) - [Learning Decentralized Flocking Controllers with Spatio-Temporal Graph Neural Network](https://mlsanity.com/paper/2309.17437/) (2023) - [Sequential Decision Making with Expert Demonstrations under Unobserved Heterogeneity](https://mlsanity.com/paper/2404.07266/) (2024) - [A Bayesian Solution To The Imitation Gap](https://mlsanity.com/paper/2407.00495/) (2024) - [Exploring Causality for HRI: A Case Study on Robotic Mental Well-being Coaching](https://mlsanity.com/paper/2503.11684/) (2025) - [Distilling Realizable Students from Unrealizable Teachers](https://mlsanity.com/paper/2505.09546/) (2025) - [Safe reinforcement learning in uncertain contexts](https://mlsanity.com/paper/2401.05876/) (2024) - [Identifying Differential Patient Care Through Inverse Intent Inference](https://mlsanity.com/paper/2411.07372/) (2024) - [Causal Imitation Learning under Expert-Observable and Expert-Unobservable Confounding](https://mlsanity.com/paper/2502.07656/) (2025) - [Blindfolded Experts Generalize Better: Insights from Robotic Manipulation and Videogames](https://mlsanity.com/paper/2510.24194/) (2025) - [Solving Without Stopping: On-Policy Distillation at Small Scale](https://mlsanity.com/paper/2609.37326/) (2026) - [Overcoming Scaling Limits in On-Policy Self-Distillation for LLM Reasoning](https://mlsanity.com/paper/2609.37915/) (2026) ## Abstract (arXiv) We consider imitation learning problems where the learner's ability to mimic the expert increases throughout the course of an episode as more information is revealed. One example of this is when the expert has access to privileged information: while the learner might not be able to accurately reproduce expert behavior early on in an episode, by considering the entire history of states and actions, they might be able to eventually identify the hidden context and act as the expert would. We prove that on-policy imitation learning algorithms (with or without access to a queryable expert) are better equipped to handle these sorts of asymptotically realizable problems than off-policy methods. This is because on-policy algorithms provably learn to recover from their initially suboptimal actions, while off-policy methods treat their suboptimal past actions as though they came from the expert. This often manifests as a latching behavior: a naive repetition of past actions. We conduct experiments in a toy bandit domain that show that there exist sharp phase transitions of whether off-policy approaches are able to match expert performance asymptotically, in contrast to the uniformly good performance of on-policy approaches. We demonstrate that on several continuous control tasks, on-policy approaches are able to use history to identify the context while off-policy approaches actually perform worse when given access to history.