mlsanityData to 7 Oct 2026

Truncated Horizon Policy Search: Combining Reinforcement Learning & Imitation Learning

arXiv 1805.11240 ↗First posted 29 May 2018cs.LG stat.MLcited by 39

On the map: Offline RL and imitation learning from data → Imitation learning methods

Abstract (arXiv)

In this paper, we propose to combine imitation and reinforcement learning via the idea of reward shaping using an oracle. We study the effectiveness of the near-optimal cost-to-go oracle on the planning horizon and demonstrate that the cost-to-go oracle shortens the learner's planning horizon as function of its accuracy: a globally optimal oracle can shorten the planning horizon to one, leading to a one-step greedy Markov Decision Process which is much easier to optimize, while an oracle that is far away from the optimality requires planning over a longer horizon to achieve near-optimal performance. Hence our new insight bridges the gap and interpolates between imitation learning and reinforcement learning. Motivated by the above mentioned insights, we propose Truncated HORizon Policy Search (THOR), a method that focuses on searching for policies that maximize the total reshaped reward over a finite planning horizon when the oracle is sub-optimal. We experimentally demonstrate that a gradient-based implementation of THOR can achieve superior performance compared to RL baselines and IL baselines even when the oracle is sub-optimal.

Key facts. Cited by 39 papers in the map (data through 7 Oct 2026). Placed in Imitation learning methods, one of 13 directions in Offline RL and imitation learning from data. Builds on 14 of its 31 references that are in the map. First posted on arXiv 29 May 2018.

Builds on

14 of 31 references.

References · 14

  1. 2015Continuous control with deep reinforcement learningcited by 3,279
  2. 2015Trust Region Policy Optimizationcited by 2,503
  3. 2015High-Dimensional Continuous Control Using Generalized Advantage Estimationcited by 1,916
  4. 2016OpenAI Gymcited by 1,684
  5. 2017Learning Complex Dexterous Manipulation with Deep Reinforcement Learning and Demonstrationscited by 685
  6. 2017Overcoming Exploration in Reinforcement Learning with Demonstrationscited by 322
  7. 2017Leveraging Demonstrations for Deep Reinforcement Learning on Robotics Problems with Sparse Rewardscited by 302
  8. 2016An Actor-Critic Algorithm for Sequence Predictioncited by 158
  9. 2014Reinforcement and Imitation Learning via Interactive No-Regret Learningcited by 116
  10. 2017Deeply AggreVaTeD: Differentiable Imitation Learning for Sequential Predictioncited by 93
  11. 2015Learning to Search Better Than Your Teachercited by 50
  12. 2017Agile Autonomous Driving using End-to-End Deep Imitation Learningcited by 9
2 more
  1. 2015Learning to Filter with Predictive State Inference Machinescited by 8
  2. 2016Learning to Gather Information via Imitationcited by 4

Built on by

Cited by year

Citing papers per year: 2019: 5; 2020: 2; 2021: 6; 2022: 9; 2023: 8; 2024: 4; 2025: 5.

39 papers.

Sort:

By area:

  1. 2019Goal-conditioned Imitation Learningcited by 159
  2. 2019Imitation Learning as f-Divergence Minimizationcited by 121
  3. 2019BAIL: Best-Action Imitation Learning for Batch Deep Reinforcement Learningcited by 90
  4. 2019Provably Efficient Imitation Learning from Observation Alonecited by 78
  5. 2023Embodied Multi-Modal Agent trained by an LLM from a Parallel TextWorldcited by 65
  6. 2022Reincarnating Reinforcement Learning: Reusing Prior Computation to Accelerate Progresscited by 63
  7. 2023Learning to Generate Better Than Your LLMcited by 49
  8. 2020Provable Representation Learning for Imitation Learning via Bi-level Optimizationcited by 41
  9. 2021Heuristic-Guided Reinforcement Learningcited by 41
  10. 2019Watch, Try, Learn: Meta-Learning from Demonstrations and Rewardcited by 26
  11. 2023RLIF: Interactive Imitation Learning as Reinforcement Learningcited by 24
  12. 2022Pareto Conditioned Networkscited by 22
27 more
  1. 2025GRPO-λ: Credit Assignment improves LLM Reasoningcited by 21
  2. 2021Improving Zero-shot Generalization in Offline Reinforcement Learning using Generalized Similarity Functionscited by 17
  3. 2022Real World Offline Reinforcement Learning with Realistic Data Sourcecited by 16
  4. 2023Blending Imitation and Reinforcement Learning for Robust Policy Improvementcited by 16
  5. 2025Rapidly Adapting Policies to the Real World via Simulation-Guided Fine-Tuningcited by 15
  6. 2022Resolving Copycat Problems in Visual Imitation Learning via Residual Action Predictioncited by 14
  7. 2025DexTrack: Towards Generalizable Neural Tracking Control for Dexterous Manipulation from Human Referencescited by 13
  8. 2022MetaTrader: An Reinforcement Learning Approach Integrating Diverse Policies for Portfolio Optimizationcited by 10
  9. 2021The Value of Planning for Infinite-Horizon Model Predictive Controlcited by 8
  10. 2022Learning Graph Search Heuristicscited by 7
  11. 2023IMM: An Imitative Reinforcement Learning Approach with Predictive Representation Learning for Automatic Market Makingcited by 6
  12. 2020POPO: Pessimistic Offline Policy Optimizationcited by 5
  13. 2023Visual Hindsight Self-Imitation Learning for Interactive Navigationcited by 5
  14. 2022Combining imitation and deep reinforcement learning to accomplish human-level performance on a virtual foraging taskcited by 4
  15. 2021Improving Long-Term Metrics in Recommendation Systems using Short-Horizon Reinforcement Learningcited by 2
  16. 2025A Simple Approach to Constraint-Aware Imitation Learning with Application to Autonomous Racingcited by 2
  17. 2022Generalizing to New Tasks via One-Shot Compositional Subgoalscited by 1
  18. 2023Truncating Trajectories in Monte Carlo Reinforcement Learningcited by 1
  19. 2024Truncating Trajectories in Monte Carlo Policy Evaluation: an Adaptive Approachcited by 1
  20. 2024Interactive and Hybrid Imitation Learning: Provably Beating Behavior Cloningcited by 1
  21. 2021Reinforced Imitation Learning by Free Energy Principlecited by 0
  22. 2021Leveraging Experience in Lazy Searchcited by 0
  23. 2022Reinforcement Learning from Demonstrations by Novel Interactive Expert and Application to Automatic Berthing Control Systems for Unmanned Surface Vesselcited by 0
  24. 2023Policy Optimization with Smooth Guidance Learned from State-Only Demonstrationscited by 0
  25. 2024Accelerating Inverse Reinforcement Learning with Expert Bootstrappingcited by 0
  26. 2024Snapshot Reinforcement Learning: Leveraging Prior Trajectories for Efficiencycited by 0
  27. 2025Dynamic Action Interpolation: A Universal Approach for Accelerating Reinforcement Learning with Expert Guidancecited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2020Policy Improvement via Imitation of Multiple Oracles ↗
  2. 2025Beyond-Expert Performance with Limited Demonstrations: Efficient Imitation Learning with Double Exploration
  3. 2018Hierarchical Imitation and Reinforcement Learning
  4. 2018Fast Policy Learning through Imitation and Reinforcement
  5. 2019Reinforced Imitation in Heterogeneous Action Space
  6. 2026Learning from the Near Future: Temporal Self-Distillation for RLVR
  7. 2018Learning Self-Imitating Diverse Policies
  8. 2019Reward-Conditioned Policies