mlsanityData to 14 Oct 2026

The Courage to Stop: Overcoming Sunk Cost Fallacy in Deep Reinforcement Learning

arXiv 2506.13672 ↗First posted 16 Jun 2025cs.LGcited by 6

On the map: Deep reinforcement learning methods → Deep RL algorithms and representations

Abstract (arXiv)

Off-policy deep reinforcement learning (RL) typically leverages replay buffers for reusing past experiences during learning. This can help improve sample efficiency when the collected data is informative and aligned with the learning objectives; when that is not the case, it can have the effect of "polluting" the replay buffer with data which can exacerbate optimization challenges in addition to wasting environment interactions due to wasteful sampling. We argue that sampling these uninformative and wasteful transitions can be avoided by addressing the sunk cost fallacy, which, in the context of deep RL, is the tendency towards continuing an episode until termination. To address this, we propose learn to stop (LEAST), a lightweight mechanism that enables strategic early episode termination based on Q-value and gradient statistics, which helps agents recognize when to terminate unproductive episodes early. We demonstrate that our method improves learning efficiency on a variety of RL algorithms, evaluated on both the MuJoCo and DeepMind Control Suite benchmarks.

Key facts. Cited by 6 papers in the map (data through 14 Oct 2026). Placed in Deep RL algorithms and representations, one of 15 directions in Deep reinforcement learning methods. Builds on 48 of its 63 references that are in the map. First posted on arXiv 16 Jun 2025.

Builds on

48 of 63 references.

Closest ancestors · 5

Same direction, newest first

  1. 2024Don't flatten, tokenize! Unlocking the key to SoftMoE's efficacy in deep RL2 Oct
  2. 2024On the consistency of hyper-parameter selection in value-based deep reinforcement learning25 Jun
  3. 2024Bigger, Regularized, Optimistic: scaling for compute and sample-efficient continuous control25 May
  4. 2024In value-based deep reinforcement learning, a pruned network is a good network19 Feb
  5. 2024Mixtures of Experts Unlock Parameter Scaling for Deep RL13 Feb

Other references · 43

  1. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
  2. 2015Continuous control with deep reinforcement learningcited by 3,407
  3. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,417
  4. 2015Deep Reinforcement Learning with Double Q-learningcited by 1,745
  5. 2016OpenAI Gymcited by 1,736
  6. 2017Curiosity-driven Exploration by Self-supervised Predictioncited by 1,194
  7. 2015Prioritized Experience Replaycited by 1,099
  8. 2018DeepMind Control Suitecited by 956
  9. 2021Mastering Visual Continuous Control: Improved Data-Augmented Reinforcement Learningcited by 372
  10. 2021Randomized Ensembled Double Q-Learning: Learning Fast Without a Modelcited by 288
  11. 2022The Primacy Bias in Deep Reinforcement Learningcited by 247
  12. 2023Scaling Robot Learning with Semantically Imagined Experiencecited by 177
31 more
  1. 2020Revisiting Fundamentals of Experience Replaycited by 175
  2. 2023The Dormant Neuron Phenomenon in Deep Reinforcement Learningcited by 174
  3. 2023Bigger, Better, Faster: Human-level Atari with human-level efficiencycited by 147
  4. 2023Loss of Plasticity in Continual Deep Reinforcement Learningcited by 121
  5. 2021Dropout Q-Functions for Doubly Efficient Reinforcement Learningcited by 114
  6. 2019Optimism in Reinforcement Learning with Generalized Linear Function Approximationcited by 111
  7. 2023Synthetic Experience Replaycited by 101
  8. 2017Time Limits in Reinforcement Learningcited by 79
  9. 2020Revisiting Rainbow: Promoting more Insightful and Inclusive Deep Reinforcement Learning Researchcited by 78
  10. 2023TACO: Temporal Latent Action-Driven Contrastive Loss for Visual Reinforcement Learningcited by 57
  11. 2021Tactical Optimism and Pessimism for Deep Reinforcement Learningcited by 55
  12. 2023DrM: Mastering Visual Reinforcement Learning through Dormant Ratio Minimizationcited by 47
  13. 2023Small batch deep reinforcement learningcited by 36
  14. 2021HyAR: Addressing Discrete-Continuous Action Reinforcement Learning via Hybrid Action Representationcited by 35
  15. 2020Softmax Deep Double Deterministic Policy Gradientscited by 32
  16. 2022Stabilizing Off-Policy Deep Reinforcement Learning from Pixelscited by 32
  17. 2021Medical Dead-ends and Learning to Identify High-risk States and Treatmentscited by 28
  18. 2023Planning Goals for Explorationcited by 28
  19. 2023Revisiting Plasticity in Visual Reinforcement Learning: Data, Modules and Training Stagescited by 28
  20. 2020Optimistic Exploration even with a Pessimistic Initialisationcited by 25
  21. 2022Prioritizing Samples in Reinforcement Learning with Reducible Losscited by 22
  22. 2021Regularized Softmax Deep Multi-Agent Q-Learningcited by 19
  23. 2021Safe Exploration by Solving Early Terminated MDPcited by 15
  24. 2019Reinforcement Learning with Dynamic Boltzmann Softmax Updatescited by 14
  25. 2022Model-Free Generative Replay for Lifelong Reinforcement Learning: Application to Starcraft-2cited by 11
  26. 2023Sample Efficient Deep Reinforcement Learning via Local Planningcited by 10
  27. 2017Enhanced Experience Replay Generation for Efficient Reinforcement Learningcited by 6
  28. 2023Risk Sensitive Dead-end Identification in Safety-Critical Offline Reinforcement Learningcited by 6
  29. 2023Online Reinforcement Learning with Uncertain Episode Lengthscited by 4
  30. 2023Truncating Trajectories in Monte Carlo Reinforcement Learningcited by 1
  31. 2024Truncating Trajectories in Monte Carlo Policy Evaluation: an Adaptive Approachcited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 3; 2026: 3.

6 papers.

  1. 2025Stable Gradients for Stable Learning at Scale in Deep Reinforcement Learningcited by 23
  2. 2025Simplicial Embeddings Improve Sample Efficiency in Actor-Critic Agentscited by 12
  3. 2025Asymmetric Proximal Policy Optimization: mini-critics boost LLM reasoningcited by 10
  4. 2026Stable Deep Reinforcement Learning via Isotropic Gaussian Representationscited by 5
  5. 2026When are LLMs Sufficient Policy Optimizers for Sequential RL Tasks?cited by 0
  6. 2026EBRL: Asynchronous Embodied RL by Multi-Grained Resource Managementcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Turning Sand to Gold: Recycling Data to Bridge On-Policy and Off-Policy Learning via Causal Bound
  2. 2026Replay What Matters: Off-Policy Replay for Efficient LLM Reinforcement Unlearning
  3. 2026Endpoint Replay: Compressing the Recency Buffer in Deep Reinforcement Learning
  4. 2025Stabilizing Reinforcement Learning with LLMs: Formulation and Practices
  5. 2026WarpSAC: Towards the Pinnacle of Scalable Off-policy RL by Rethinking Exploration and Exploitation
  6. 2024Frugal Actor-Critic: Sample Efficient Off-Policy Deep Reinforcement Learning Using Unique Experiences
  7. 2018Learning Self-Imitating Diverse Policies
  8. 2024CUER: Corrected Uniform Experience Replay for Off-Policy Continuous Deep Reinforcement Learning Algorithms