# Learning to Play in a Day: Faster Deep Reinforcement Learning by Optimality Tightening

arXiv: https://arxiv.org/abs/1611.01606 · HTML: https://mlsanity.com/paper/1611.01606/

## Key facts

- Cited by 21 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 5 Nov 2016.

## Built on by (most cited first, 21 of 21)

- [Self-Imitation Learning](https://mlsanity.com/paper/1806.05635/) (2018)
- [Human-Level Reinforcement Learning through Theory-Based Modeling, Exploration, and Planning](https://mlsanity.com/paper/2107.12544/) (2021)
- [Generalizable Episodic Memory for Deep Reinforcement Learning](https://mlsanity.com/paper/2103.06469/) (2021)
- [Regularized Softmax Deep Multi-Agent Q-Learning](https://mlsanity.com/paper/2103.11883/) (2021)
- [Constrained unsupervised anomaly segmentation](https://mlsanity.com/paper/2203.01671/) (2022)
- [Test-Time Adaptation with Shape Moments for Image Segmentation](https://mlsanity.com/paper/2205.07983/) (2022)
- [B-Coder: Value-Based Deep Reinforcement Learning for Program Synthesis](https://mlsanity.com/paper/2310.03173/) (2023)
- [Model-Based Episodic Memory Induces Dynamic Hybrid Controls](https://mlsanity.com/paper/2111.02104/) (2021)
- [Self-Imitation Learning via Generalized Lower Bound Q-learning](https://mlsanity.com/paper/2006.07442/) (2020)
- [Weakly Coupled Deep Q-Networks](https://mlsanity.com/paper/2310.18803/) (2023)
- [Looking at the whole picture: constrained unsupervised anomaly segmentation](https://mlsanity.com/paper/2109.00482/) (2021)
- [Boosting Soft Q-Learning by Bounding](https://mlsanity.com/paper/2406.18033/) (2024)
- [Estimation Error Correction in Deep Reinforcement Learning for Deterministic Actor-Critic Methods](https://mlsanity.com/paper/2109.10736/) (2021)
- [MAN: Multi-Action Networks Learning](https://mlsanity.com/paper/2209.09329/) (2022)
- [Highway Reinforcement Learning](https://mlsanity.com/paper/2405.18289/) (2024)
- [Efficient Sparse-Reward Goal-Conditioned Reinforcement Learning with a High Replay Ratio and Regularization](https://mlsanity.com/paper/2312.05787/) (2023)
- [Knowing the Past to Predict the Future: Reinforcement Virtual Learning](https://mlsanity.com/paper/2211.01266/) (2022)
- [Efficient Offline Reinforcement Learning: First Imitate, then Improve](https://mlsanity.com/paper/2406.13376/) (2024)
- [Long-Horizon Q-Learning: Accurate Value Learning via n-Step Inequalities](https://mlsanity.com/paper/2605.05812/) (2026)
- [Upper-Expectile Multi-Step Q-Learning for Off-Policy Reinforcement Learning](https://mlsanity.com/paper/2608.02034/) (2026)
- [Lifted Bellman Linear Programming for Offline Reinforcement Learning](https://mlsanity.com/paper/2609.24489/) (2026)

## Abstract (arXiv)

We propose a novel training algorithm for reinforcement learning which combines the strength of deep Q-learning with a constrained optimization approach to tighten optimality and encourage faster reward propagation. Our novel technique makes deep reinforcement learning more practical by drastically reducing the training time. We evaluate the performance of our approach on the 49 games of the challenging Arcade Learning Environment, and report significant improvements in both training time and accuracy.