mlsanityData to 14 Oct 2026

Time Limits in Reinforcement Learning

arXiv 1712.00378 ↗First posted 1 Dec 2017cs.LGcited by 79

Not on the map. Why?

Cited by 30 in Deep reinforcement learning methods, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

In reinforcement learning, it is common to let an agent interact for a fixed amount of time with its environment before resetting it and repeating the process in a series of episodes. The task that the agent has to learn can either be to maximize its performance over (i) that fixed period, or (ii) an indefinite period where time limits are only used during training to diversify experience. In this paper, we provide a formal account for how time limits could effectively be handled in each of the two cases and explain why not doing so can cause state aliasing and invalidation of experience replay, leading to suboptimal policies and training instability. In case (i), we argue that the terminations due to time limits are in fact part of the environment, and thus a notion of the remaining time should be included as part of the agent's input to avoid violation of the Markov property. In case (ii), the time limits are not part of the environment and are only used to facilitate learning. We argue that this insight should be incorporated by bootstrapping from the value of the state at the end of each partial episode. For both cases, we illustrate empirically the significance of our considerations in improving the performance and stability of existing reinforcement learning algorithms, showing state-of-the-art results on several control tasks.

Key facts. Cited by 79 papers in the map (data through 14 Oct 2026). First posted on arXiv 1 Dec 2017.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2020: 1; 2021: 15; 2022: 8; 2023: 13; 2024: 18; 2025: 7; 2026: 17.

79 papers.

Sort:

By area:

  1. 2021Learning to Walk in Minutes Using Massively Parallel Deep Reinforcement Learningcited by 512
  2. 2021Tianshou: a Highly Modularized Deep Reinforcement Learning Librarycited by 143
  3. 2021Learning to Iteratively Solve Routing Problems with Dual-Aspect Collaborative Transformercited by 108
  4. 2021d3rlpy: An Offline Deep Reinforcement Learning Librarycited by 84
  5. 2021Greedification Operators for Policy Optimization: Investigating Forward and Reverse KL Divergencescited by 41
  6. 2023AMAGO: Scalable In-Context Reinforcement Learning for Adaptive Agentscited by 41
  7. 2024EgoGen: An Egocentric Synthetic Data Generatorcited by 27
  8. 2024No Representation, No Trust: Connecting Representation, Collapse, and Trust Issues in PPOcited by 27
  9. 2024Streaming Deep Reinforcement Learning Finally Workscited by 25
  10. 2022The Free Energy Principle for Perception and Action: A Deep Learning Perspectivecited by 24
  11. 2023Beyond Uniform Sampling: Offline Reinforcement Learning with Imbalanced Datasetscited by 24
  12. 2023ZSC-Eval: An Evaluation Toolkit and Benchmark for Multi-agent Zero-shot Coordinationcited by 24
38 more
  1. 2021Time Discretization-Invariant Safe Action Repetition for Policy Gradient Methodscited by 23
  2. 2023Benchmarking Offline Reinforcement Learning on Real-Robot Hardwarecited by 23
  3. 2024Fine-tuning Reinforcement Learning Models is Secretly a Forgetting Mitigation Problemcited by 23
  4. 2021Robust High-speed Running for Quadruped Robots via Deep Reinforcement Learningcited by 22
  5. 2022N2M2: Learning Navigation for Arbitrary Mobile Manipulation Motions in Unseen and Dynamic Environmentscited by 16
  6. 2024A Reductions Approach to Risk-Sensitive Reinforcement Learning with Optimized Certainty Equivalentscited by 16
  7. 2020Lipschitzness Is All You Need To Tame Off-policy Generative Adversarial Imitation Learningcited by 15
  8. 2021Safe Exploration by Solving Early Terminated MDPcited by 15
  9. 2025Evolution and The Knightian Blindspot of Machine Learningcited by 11
  10. 2021OstrichRL: A Musculoskeletal Ostrich Simulation to Study Bio-mechanical Locomotioncited by 10
  11. 2021A Pragmatic Look at Deep Imitation Learningcited by 9
  12. 2021An Alternate Policy Gradient Estimator for Softmax Policiescited by 9
  13. 2021Examining average and discounted reward optimality criteria in reinforcement learningcited by 6
  14. 2022AdaTest:Reinforcement Learning and Adaptive Sampling for On-chip Hardware Trojan Detectioncited by 6
  15. 2023Fair collaborative vehicle routing: A deep multi-agent reinforcement learning approachcited by 6
  16. 2024Multi-Agent Reinforcement Learning Meets Leaf Sequencing in Radiotherapycited by 6
  17. 2025The Courage to Stop: Overcoming Sunk Cost Fallacy in Deep Reinforcement Learningcited by 6
  18. 2021Towards Automatic Actor-Critic Solutions to Continuous Controlcited by 5
  19. 2023Intentionally-underestimated Value Function at Terminal State for Temporal-difference Learning with Mis-designed Rewardcited by 5
  20. 2023An Invitation to Deep Reinforcement Learningcited by 5
  21. 2024Deployable Reinforcement Learning with Variable Control Ratecited by 5
  22. 2024Population-aware Online Mirror Descent for Mean-Field Games by Deep Reinforcement Learningcited by 5
  23. 2024Scalable Reinforcement Learning-based Neural Architecture Searchcited by 5
  24. 2025Emergent Risk Awareness in Rational Agents under Resource Constraintscited by 5
  25. 2021Taylor Expansion of Discount Factorscited by 4
  26. 2024Intelligent Switching for Reset-Free RLcited by 4
  27. 2024Reinforcement Learning with Elastic Time Stepscited by 3
  28. 2024Reciprocal Reward Influence Encourages Cooperation From Self-Interested Agentscited by 3
  29. 2022DNA: Proximal Policy Optimization with a Dual Network Architecturecited by 2
  30. 2022D-Shape: Demonstration-Shaped Reinforcement Learning via Goal Conditioningcited by 2
  31. 2022Design and Planning of Flexible Mobile Micro-Grids Using Deep Reinforcement Learningcited by 2
  32. 2023Multi-Agent Reinforcement Learning for Fast-Timescale Demand Response of Residential Loadscited by 2
  33. 2023Parallel bootstrap-based on-policy deep reinforcement learning for continuous flow control applicationscited by 2
  34. 2023Optimizing Attention and Cognitive Control Costs Using Temporally-Layered Architecturescited by 2
  35. 2023Non-Stationary Policy Learning for Multi-Timescale Multi-Agent Reinforcement Learningcited by 2
  36. 2021AWD3: Dynamic Reduction of the Estimation Biascited by 1
  37. 2023MARLIN: Soft Actor-Critic based Reinforcement Learning for Congestion Control in Real Networkscited by 1
  38. 2024FREED++: Improving RL Agents for Fragment-Based Molecule Generation by Thorough Reproductioncited by 1

50 most cited of 79 All 79 (JSON)