mlsanityData to 7 Oct 2026

Learning Montezuma's Revenge from a Single Demonstration

arXiv 1812.03381 ↗First posted 8 Dec 2018cs.LG cs.AI cs.NEcited by 76

Not on the map. Why?

Abstract (arXiv)

We propose a new method for learning from a single demonstration to solve hard exploration tasks like the Atari game Montezuma's Revenge. Instead of imitating human demonstrations, as proposed in other recent works, our approach is to maximize rewards directly. Our agent is trained using off-the-shelf reinforcement learning, but starts every episode by resetting to a state from a demonstration. By starting from such demonstration states, the agent requires much less exploration to learn a game compared to when it starts from the beginning of the game at every episode. We analyze reinforcement learning for tasks with sparse rewards in a simple toy environment, where we show that the run-time of standard RL methods scales exponentially in the number of states between rewards. Our method reduces this to quadratic scaling, opening up many tasks that were previously infeasible. We then apply our method to Montezuma's Revenge, for which we present a trained agent achieving a high-score of 74,500, better than any previously published result.

Key facts. Cited by 76 papers in the map (data through 7 Oct 2026). First posted on arXiv 8 Dec 2018.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2019: 1; 2020: 0; 2021: 14; 2022: 11; 2023: 13; 2024: 14; 2025: 11; 2026: 12.

76 papers.

Sort:

By area:

  1. 2021Bridging Offline Reinforcement Learning and Imitation Learning: A Tale of Pessimismcited by 275
  2. 2022Exploration in Deep Reinforcement Learning: A Surveycited by 158
  3. 2024Teaching Large Language Models to Reason with Reinforcement Learningcited by 158
  4. 2022Jump-Start Reinforcement Learningcited by 102
  5. 2024StepCoder: Improve Code Generation with Reinforcement Learning from Compiler Feedbackcited by 74
  6. 2021Asymmetric self-play for automatic goal discovery in robotic manipulationcited by 71
  7. 2024Training Large Language Models for Reasoning through Reverse Curriculum Reinforcement Learningcited by 63
  8. 2021Exploration in Deep Reinforcement Learning: From Single-Agent to Multiagent Domaincited by 57
  9. 2021Imitation Learning: Progress, Taxonomies and Challengescited by 56
  10. 2023Learning to Generate Better Than Your LLMcited by 49
  11. 2024Dataset Reset Policy Optimization for RLHFcited by 41
  12. 2023Generalization to New Sequential Decision Making Tasks with In-Context Learningcited by 39
38 more
  1. 2025STeCa: Step-level Trajectory Calibration for LLM Agent Learningcited by 37
  2. 2024A Study of Plasticity Loss in On-Policy Deep Reinforcement Learningcited by 26
  3. 2025Learning to Reason at the Frontier of Learnabilitycited by 20
  4. 2022Reinforcement Learning for Branch-and-Bound Optimisation using Retrospective Trajectoriescited by 19
  5. 2023Proximal Curriculum for Reinforcement Learning Agentscited by 19
  6. 2021TeachMyAgent: a Benchmark for Automatic Curriculum Learning in Deep RLcited by 18
  7. 2023One ACT Play: Single Demonstration Behavior Cloning with Action Chunking Transformerscited by 17
  8. 2026Reuse your FLOPs: Scaling RL on Hard Problems by Conditioning on Very Off-Policy Prefixescited by 16
  9. 2025RL for Reasoning by Adaptively Revealing Rationalescited by 15
  10. 2021Wish you were here: Hindsight Goal Selection for long-horizon dexterous manipulationcited by 13
  11. 2021Provably Breaking the Quadratic Error Compounding Barrier in Imitation Learning, Optimallycited by 12
  12. 2024Reverse Forward Curriculum Learning for Extreme Sample and Demonstration Efficiency in Reinforcement Learningcited by 12
  13. 2025Reinforcement Learning Teachers of Test Time Scalingcited by 10
  14. 2023Sample Efficient Deep Reinforcement Learning via Local Planningcited by 9
  15. 2025From Reasoning to Super-Intelligence: A Search-Theoretic Perspectivecited by 9
  16. 2022Time-Efficient Reward Learning via Visually Assisted Cluster Rankingcited by 8
  17. 2023Finding Increasingly Large Extremal Graphs with AlphaZero and Tabu Searchcited by 8
  18. 2021Finding Failures in High-Fidelity Simulation using Adaptive Stress Testing and the Backward Algorithmcited by 7
  19. 2022Generative Personas That Behave and Experience Like Humanscited by 7
  20. 2024DemoStart: Demonstration-led auto-curriculum applied to sim-to-real with multi-fingered robotscited by 7
  21. 2021Go-Blend behavior and affectcited by 6
  22. 2022Leveraging Sequentiality in Reinforcement Learning from a Single Demonstrationcited by 6
  23. 2024Proximal Curriculum with Task Correlations for Deep Reinforcement Learningcited by 6
  24. 2025Causally Aligned Curriculum Learningcited by 6
  25. 2019ZPD Teaching Strategies for Deep Reinforcement Learning from Demonstrationscited by 5
  26. 2021Plan-Based Relaxed Reward Shaping for Goal-Directed Taskscited by 5
  27. 2021Learning from Ambiguous Demonstrations with Self-Explanation Guided Reinforcement Learningcited by 5
  28. 2023Contrastive Initial State Buffer for Reinforcement Learningcited by 5
  29. 2024Hybrid Reinforcement Learning from Offline Observation Alonecited by 5
  30. 2021Automatic Curricula via Expert Demonstrationscited by 4
  31. 2021Learning Multi-Objective Curricula for Robotic Policy Learningcited by 3
  32. 2022Towards Improving Exploration in Self-Imitation Learning using Intrinsic Motivationcited by 3
  33. 2024Unsupervised Skill Discovery for Robotic Manipulation through Automatic Task Generationcited by 3
  34. 2022Divide & Conquer Imitation Learningcited by 2
  35. 2022Task Phasing: Automated Curriculum Learning from Demonstrationscited by 2
  36. 2022D-Shape: Demonstration-Shaped Reinforcement Learning via Goal Conditioningcited by 2
  37. 2023CRISP: Curriculum Inducing Primitive Informed Subgoal Prediction for Hierarchical Reinforcement Learningcited by 2
  38. 2024WayEx: Waypoint Exploration using a Single Demonstrationcited by 2

50 most cited of 76 All 76 (JSON)