# Playing Atari Games with Deep Reinforcement Learning and Human Checkpoint Replay

arXiv: https://arxiv.org/abs/1607.05077 · HTML: https://mlsanity.com/paper/1607.05077/

## Key facts

- Cited by 22 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 18 Jul 2016.

## Built on by (most cited first, 22 of 22)

- [Jump-Start Reinforcement Learning](https://mlsanity.com/paper/2204.02372/) (2022)
- [StepCoder: Improve Code Generation with Reinforcement Learning from Compiler Feedback](https://mlsanity.com/paper/2402.01391/) (2024)
- [Training Large Language Models for Reasoning through Reverse Curriculum Reinforcement Learning](https://mlsanity.com/paper/2402.05808/) (2024)
- [Option-Aware Adversarial Inverse Reinforcement Learning for Robotic Control](https://mlsanity.com/paper/2210.01969/) (2022)
- [Reverse Forward Curriculum Learning for Extreme Sample and Demonstration Efficiency in Reinforcement Learning](https://mlsanity.com/paper/2405.03379/) (2024)
- [Learning to Shape Rewards using a Game of Two Partners](https://mlsanity.com/paper/2103.09159/) (2021)
- [Finding Increasingly Large Extremal Graphs with AlphaZero and Tabu Search](https://mlsanity.com/paper/2311.03583/) (2023)
- [A Unified Algorithm Framework for Unsupervised Discovery of Skills based on Determinantal Point Process](https://mlsanity.com/paper/2212.00211/) (2022)
- [Training Interactive Agent in Large FPS Game Map with Rule-enhanced Reinforcement Learning](https://mlsanity.com/paper/2410.04936/) (2024)
- [Learning Multi-agent Skills for Tabular Reinforcement Learning using Factor Graphs](https://mlsanity.com/paper/2201.08227/) (2022)
- [Deep Q-Network Based Decision Making for Autonomous Driving](https://mlsanity.com/paper/2303.11634/) (2023)
- [WIMLE: Uncertainty-Aware World Models with IMLE for Sample-Efficient Continuous Control](https://mlsanity.com/paper/2602.14351/) (2026)
- [Reinforcement Learning For Constraint Satisfaction Game Agents (15-Puzzle, Minesweeper, 2048, and Sudoku)](https://mlsanity.com/paper/2102.06019/) (2021)
- [Boosting Reinforcement Learning and Planning with Demonstrations: A Survey](https://mlsanity.com/paper/2303.13489/) (2023)
- [Model-enhanced Contrastive Reinforcement Learning for Sequential Recommendation](https://mlsanity.com/paper/2310.16566/) (2023)
- [Verifying Memoryless Sequential Decision-making of Large Language Models](https://mlsanity.com/paper/2510.06756/) (2025)
- [Learning from Demonstrations via Capability-Aware Goal Sampling](https://mlsanity.com/paper/2601.08731/) (2026)
- [Joint Learning of Hierarchical Neural Options and Abstract World Model](https://mlsanity.com/paper/2602.02799/) (2026)
- [Accelerating Inverse Reinforcement Learning with Expert Bootstrapping](https://mlsanity.com/paper/2402.02608/) (2024)
- [Snapshot Reinforcement Learning: Leveraging Prior Trajectories for Efficiency](https://mlsanity.com/paper/2403.00673/) (2024)
- [Imitation Learning of Correlated Policies in Stackelberg Games](https://mlsanity.com/paper/2503.08883/) (2025)
- [Going Beyond Expert Performance via Deep Implicit Imitation Reinforcement Learning](https://mlsanity.com/paper/2511.03616/) (2025)

## Abstract (arXiv)

This paper introduces a novel method for learning how to play the most difficult Atari 2600 games from the Arcade Learning Environment using deep reinforcement learning. The proposed method, human checkpoint replay, consists in using checkpoints sampled from human gameplay as starting points for the learning process. This is meant to compensate for the difficulties of current exploration strategies, such as epsilon-greedy, to find successful control policies in games with sparse rewards. Like other deep reinforcement learning architectures, our model uses a convolutional neural network that receives only raw pixel inputs to estimate the state value function. We tested our method on Montezuma's Revenge and Private Eye, two of the most challenging games from the Atari platform. The results we obtained show a substantial improvement compared to previous learning approaches, as well as over a random player. We also propose a method for training deep reinforcement learning agents using human gameplay experience, which we call human experience replay.