mlsanityData to 14 Oct 2026

2048: Reinforcement Learning in a Delayed Reward Environment

arXiv 2507.05465 ↗First posted 7 Jul 2025cs.LG cs.AIcited by 0

On the map: Deep reinforcement learning methods → Deep RL algorithms and representations

Abstract (arXiv)

Delayed and sparse rewards present a fundamental obstacle for reinforcement-learning (RL) agents, which struggle to assign credit for actions whose benefits emerge many steps later. The sliding-tile game 2048 epitomizes this challenge: although frequent small score changes yield immediate feedback, they often mislead agents into locally optimal but globally suboptimal strategies. In this work, we introduce a unified, distributional multi-step RL framework designed to directly optimize long-horizon performance. Using the open source Gym-2048 environment we develop and compare four agent variants: standard DQN, PPO, QR-DQN (Quantile Regression DQN), and a novel Horizon-DQN (H-DQN) that integrates distributional learning, dueling architectures, noisy networks, prioritized replay, and more. Empirical evaluation reveals a clear hierarchy in effectiveness: max episode scores improve from 3.988K (DQN) to 5.756K (PPO), 8.66K (QR-DQN), and 18.21K (H-DQN), with H-DQN reaching the 2048 tile. Upon scaling H-DQN it reaches a max score 41.828K and a 4096 tile. These results demonstrate that distributional, multi-step targets substantially enhance performance in sparse-reward domains, and they suggest promising avenues for further gains through model-based planning and curriculum learning.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Deep RL algorithms and representations, one of 15 directions in Deep reinforcement learning methods. Builds on 33 of its 49 references that are in the map. First posted on arXiv 7 Jul 2025.

Builds on

33 of 49 references.

References · 33

  1. 2017Proximal Policy Optimization Algorithmscited by 15,215
  2. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
  3. 2019Optuna: A Next-generation Hyperparameter Optimization Frameworkcited by 2,501
  4. 2016Asynchronous Methods for Deep Reinforcement Learningcited by 2,258
  5. 2015High-Dimensional Continuous Control Using Generalized Advantage Estimationcited by 2,026
  6. 2015Deep Reinforcement Learning with Double Q-learningcited by 1,745
  7. 2019Mastering Atari, Go, Chess and Shogi by Planning with a Learned Modelcited by 1,353
  8. 2017Curiosity-driven Exploration by Self-supervised Predictioncited by 1,194
  9. 2017Hindsight Experience Replaycited by 1,136
  10. 2015Prioritized Experience Replaycited by 1,099
  11. 2018Exploration by Random Network Distillationcited by 993
  12. 2018IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architecturescited by 883
21 more
  1. 2017Rainbow: Combining Improvements in Deep Reinforcement Learningcited by 786
  2. 2015Dueling Network Architectures for Deep Reinforcement Learningcited by 785
  3. 2017A Distributional Perspective on Reinforcement Learningcited by 765
  4. 2020Deep Reinforcement Learning for Autonomous Driving: A Surveycited by 721
  5. 2016Deep Exploration via Bootstrapped DQNcited by 527
  6. 2017Distributional Reinforcement Learning with Quantile Regressioncited by 474
  7. 2016Reinforcement Learning with Unsupervised Auxiliary Taskscited by 404
  8. 2018Implicit Quantile Networks for Distributional Reinforcement Learningcited by 369
  9. 2017PathNet: Evolution Channels Gradient Descent in Super Neural Networkscited by 364
  10. 2020Critic Regularized Regressioncited by 275
  11. 2018Distributed Prioritized Experience Replaycited by 269
  12. 2017Noisy Networks for Explorationcited by 266
  13. 2020First return, then explorecited by 266
  14. 2020Revisiting Fundamentals of Experience Replaycited by 175
  15. 2018Meta-Gradient Reinforcement Learningcited by 159
  16. 2018Multi-task Deep Reinforcement Learning with PopArtcited by 141
  17. 2017Imagination-Augmented Agents for Deep Reinforcement Learningcited by 140
  18. 2018Deep Hedgingcited by 108
  19. 2020Munchausen Reinforcement Learningcited by 69
  20. 2019An investigation of model-free planningcited by 66
  21. 2021Optimistic Temporal Difference Learning for 2048cited by 2

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026MindGames Arena Generalization Track: In2AI Solution with Delayed Per-Step Reward Attribution
  2. 2026Neglected Free Lunch from Post-training: Progress Advantage for LLM Agents
  3. 2026Reinforcement Learning: From Algorithms To Foundation Models
  4. 2024Beyond Simple Sum of Delayed Rewards: Non-Markovian Reward Modeling for Reinforcement Learning
  5. 2025SPA-RL: Reinforcing LLM Agents via Stepwise Progress Attribution
  6. 2025A Differential Perspective on Distributional Reinforcement Learning
  7. 2024The Distributional Reward Critic Framework for Reinforcement Learning Under Perturbed Rewards
  8. 2025DISCOVER: Automated Curricula for Sparse-Reward Reinforcement Learning