mlsanityData to 14 Oct 2026

A Smooth Sea Never Made a Skilled SAILOR: Robust Imitation via Learning to Search

arXiv 2506.05294 ↗First posted 5 Jun 2025cs.LGcited by 17

On the map: Offline RL and imitation learning from data → Imitation learning methods

Abstract (arXiv)

The fundamental limitation of the behavioral cloning (BC) approach to imitation learning is that it only teaches an agent what the expert did at states the expert visited. This means that when a BC agent makes a mistake which takes them out of the support of the demonstrations, they often don't know how to recover from it. In this sense, BC is akin to giving the agent the fish -- giving them dense supervision across a narrow set of states -- rather than teaching them to fish: to be able to reason independently about achieving the expert's outcome even when faced with unseen situations at test-time. In response, we explore learning to search (L2S) from expert demonstrations, i.e. learning the components required to, at test time, plan to match expert outcomes, even after making a mistake. These include (1) a world model and (2) a reward model. We carefully ablate the set of algorithmic and design decisions required to combine these and other components for stable and sample/interaction-efficient learning of recovery behavior without additional human corrections. Across a dozen visual manipulation tasks from three benchmarks, our approach SAILOR consistently out-performs state-of-the-art Diffusion Policies trained via BC on the same data. Furthermore, scaling up the amount of demonstrations used for BC by 5-10x still leaves a performance gap. We find that SAILOR can identify nuanced failures and is robust to reward hacking. Our code is available at https://github.com/arnavkj1995/SAILOR .

Key facts. Cited by 17 papers in the map (data through 14 Oct 2026). Placed in Imitation learning methods, one of 13 directions in Offline RL and imitation learning from data. Builds on 67 of its 90 references that are in the map. First posted on arXiv 5 Jun 2025.

Builds on

67 of 90 references.

Closest ancestors · 8

Same direction, newest first

  1. 2025Efficient Imitation under Misspecification17 Mar
  2. 2024Policy Decorator: Model-Agnostic Online Refinement for Large Policy Model18 Dec
  3. 2024Diffusing States and Matching Scores: A New Framework for Imitation Learning17 Oct
  4. 2022Sequence Model Imitation Learning with Unobserved Contexts3 Aug
  5. 2022Watch and Match: Supercharging Imitation with Regularized Optimal Transport30 Jun
  6. 2022Minimax Optimal Online Imitation Learning via Replay Estimation30 May
  7. 2021Of Moments and Matching: A Game-Theoretic Framework for Closing the Imitation Gap4 Mar
  8. 2018HG-DAgger: Interactive Imitation Learning with Human Experts5 Oct

Other references · 59

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2020Denoising Diffusion Probabilistic Modelscited by 21,076
  3. 2022Training language models to follow instructions with human feedbackcited by 15,971
  4. 2020Denoising Diffusion Implicit Modelscited by 9,781
  5. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 6,330
  6. 2021Improved Denoising Diffusion Probabilistic Modelscited by 3,761
  7. 2024OpenVLA: An Open-Source Vision-Language-Action Modelcited by 2,887
  8. 2023Diffusion Policy: Visuomotor Policy Learning via Action Diffusioncited by 2,838
  9. 2020Learning to summarize from human feedbackcited by 2,588
  10. 2017Improved Training of Wasserstein GANscited by 2,235
  11. 2013Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computationcited by 2,079
  12. 2025π0.5: a Vision-Language-Action Model with Open-World Generalizationcited by 1,761
38 more
  1. 2019Dream to Control: Learning Behaviors by Latent Imaginationcited by 1,653
  2. 2016Generative Adversarial Imitation Learningcited by 1,597
  3. 2023Learning Fine-Grained Bimanual Manipulation with Low-Cost Hardwarecited by 1,568
  4. 2018Group Normalizationcited by 1,524
  5. 2018World Modelscited by 1,513
  6. 2024Octo: An Open-Source Generalist Robot Policycited by 1,484
  7. 2018Learning Latent Dynamics for Planning from Pixelscited by 1,264
  8. 2023Mastering Diverse Domains through World Modelscited by 1,253
  9. 2022Scaling Laws for Reward Model Overoptimizationcited by 1,095
  10. 2020Mastering Atari with Discrete World Modelscited by 1,007
  11. 2024Large Language Monkeys: Scaling Inference Compute with Repeated Samplingcited by 991
  12. 2022STaR: Bootstrapping Reasoning With Reasoningcited by 940
  13. 2025Cosmos World Foundation Model Platform for Physical AIcited by 864
  14. 2024DROID: A Large-Scale In-The-Wild Robot Manipulation Datasetcited by 852
  15. 2024Genie: Generative Interactive Environmentscited by 747
  16. 2021What Matters in Learning from Offline Human Demonstrations for Robot Manipulationcited by 725
  17. 2020robosuite: A Modular Simulation Framework and Benchmark for Robot Learningcited by 511
  18. 2022DayDreamer: World Models for Physical Robot Learningcited by 400
  19. 2022Temporal Difference Learning for Model Predictive Controlcited by 397
  20. 2024Universal Manipulation Interface: In-The-Wild Robot Teaching Without In-The-Wild Robotscited by 394
  21. 2024DINO-WM: World Models on Pre-trained Visual Features enable Zero-shot Planningcited by 386
  22. 2023Efficient Online Reinforcement Learning with Offline Datacited by 363
  23. 2021Randomized Ensembled Double Q-Learning: Learning Fast Without a Modelcited by 288
  24. 2024Diffusion Policy Policy Optimizationcited by 280
  25. 2017Thinking Fast and Slow with Deep Learning and Tree Searchcited by 252
  26. 2024ManiSkill3: GPU Parallelized Robotics Simulation and Rendering for Generalizable Embodied AIcited by 229
  27. 2024V-STaR: Training Verifiers for Self-Taught Reasonerscited by 217
  28. 2023GELLO: A General, Low-Cost, and Intuitive Teleoperation Framework for Robot Manipulatorscited by 189
  29. 2018Residual Policy Learningcited by 164
  30. 2024Stream of Search (SoS): Learning to Search in Languagecited by 145
  31. 2021Discovering and Achieving Goals via World Modelscited by 131
  32. 2014Reinforcement and Imitation Learning via Interactive No-Regret Learningcited by 118
  33. 2023Teach a Robot to FISH: Versatile Imitation from One Minute of Demonstrationscited by 75
  34. 2024The Ingredients for Robotic Diffusion Transformerscited by 73
  35. 2025All Roads Lead to Likelihood: The Value of Reinforcement Learning in Fine-Tuningcited by 59
  36. 2025From Foresight to Forethought: VLM-In-the-Loop Policy Steering via Latent Alignmentcited by 52
  37. 2015Learning to Search Better Than Your Teachercited by 50
  38. 2023Learning to Generate Better Than Your LLMcited by 49

50 most cited of 59 All 59 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 3; 2026: 14.

17 papers.

Sort:

By area:

  1. 2026Cosmos Policy: Fine-Tuning Video Models for Visuomotor Control and Planningcited by 300
  2. 2026DreamDojo: A Generalist Robot World Model from Large-Scale Human Videoscited by 135
  3. 2026Robometer: Scaling General-Purpose Robotic Reward Models via Trajectory Comparisonscited by 60
  4. 2026World Action Verifier: Self-Improving World Models via Forward-Inverse Asymmetrycited by 18
  5. 2026VLA-Corrector: Lightweight Detect-and-Correct Inference for Adaptive Action Horizoncited by 15
  6. 2026World2Act: Latent Action Post-Training from World Model Dynamicscited by 8
  7. 2026FAR: Failure-Aware Retry for Test-Time Recovery and Continual Policy Improvementcited by 5
  8. 2026Seeing Farther and Smarter: Value-Guided Multi-Path Reflection for VLM Policy Optimizationcited by 4
  9. 2026Learning What to Say to Your VLA: Mostly Harmless Vision Language Action Model Steeringcited by 4
  10. 2026RedFlow: Redirect Failure into Action-Level Corrections for Flow-matching VLA Policycited by 4
  11. 2025Generative Models From and For Sampling-Based MPC: A Bootstrapped Approach For Adaptive Contact-Rich Manipulationcited by 3
  12. 2025EVE: A Generator-Verifier System for Generative Policiescited by 3
5 more
  1. 2026RARM: Confidence-Gated Progress Reward Modeling for RL in Manipulationcited by 3
  2. 2025Rate optimal learning of equilibria from datacited by 2
  3. 2026Q-Learning With World Modelscited by 2
  4. 2026Online Self-Training for Co-Adaptation in Hierarchical Diffusion Policiescited by 1
  5. 2026NEEDLEWORK: Offline Rewriting of Robot Data with Verified Local Stitchescited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2020Imitating Unknown Policies via Exploration
  2. 2026Difference-Aware Retrieval Policies for Imitation Learning
  3. 2022Imitating, Fast and Slow: Robust learning from demonstrations via decision-time planning
  4. 2024Is Behavior Cloning All You Need? Understanding Horizon in Imitation Learning
  5. 2022Learning and Retrieval from Prior Data for Skill-based Imitation Learning
  6. 2026Learning When to Stop: Selective Imitation Learning Under Arbitrary Dynamics Shift
  7. 2020Offline Learning from Demonstrations and Unlabeled Experience
  8. 2018Behavioral Cloning from Observation