mlsanityData to 14 Oct 2026

Fast TRAC: A Parameter-Free Optimizer for Lifelong Reinforcement Learning

arXiv 2405.16642 ↗First posted 26 May 2024cs.LG cs.AIcited by 17

On the map: Deep reinforcement learning methods → Deep RL algorithms and representations

Abstract (arXiv)

A key challenge in lifelong reinforcement learning (RL) is the loss of plasticity, where previous learning progress hinders an agent's adaptation to new tasks. While regularization and resetting can help, they require precise hyperparameter selection at the outset and environment-dependent adjustments. Building on the principled theory of online convex optimization, we present a parameter-free optimizer for lifelong RL, called TRAC, which requires no tuning or prior knowledge about the distribution shifts. Extensive experiments on Procgen, Atari, and Gym Control environments show that TRAC works surprisingly well-mitigating loss of plasticity and rapidly adapting to challenging distribution shifts-despite the underlying optimization problem being nonconvex and nonstationary.

Key facts. Cited by 17 papers in the map (data through 14 Oct 2026). Placed in Deep RL algorithms and representations, one of 15 directions in Deep reinforcement learning methods. Builds on 43 of its 57 references that are in the map. First posted on arXiv 26 May 2024.

Builds on

43 of 57 references.

Closest ancestors · 1

Same direction, newest first

  1. 2024Disentangling the Causes of Plasticity Loss in Neural Networks29 Feb

Other references · 42

  1. 2017Proximal Policy Optimization Algorithmscited by 15,215
  2. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
  3. 2018Experience Replay for Continual Learningcited by 1,095
  4. 2018IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architecturescited by 883
  5. 2018Progress & Compress: A scalable framework for continual learningcited by 520
  6. 2019Leveraging Procedural Generation to Benchmark Reinforcement Learningcited by 483
  7. 2021Gradient Descent on Neural Networks Typically Occurs at the Edge of Stabilitycited by 427
  8. 2019Online Learning: A Modern Introduction Using Convex Optimizationcited by 417
  9. 2022The Primacy Bias in Deep Reinforcement Learningcited by 247
  10. 2023The Dormant Neuron Phenomenon in Deep Reinforcement Learningcited by 174
  11. 2023Understanding plasticity in neural networkscited by 174
  12. 2016Coin Betting and Parameter-Free Online Learningcited by 133
30 more
  1. 2018Black-Box Reductions for Parameter-free Online Learning in Banach Spacescited by 125
  2. 2022Understanding and Preventing Capacity Loss in Reinforcement Learningcited by 125
  3. 2023Loss of Plasticity in Continual Deep Reinforcement Learningcited by 121
  4. 2023A Definition of Continual Reinforcement Learningcited by 101
  5. 2021Non-stationary Reinforcement Learning without Prior Knowledge: An Optimal Black-box Approachcited by 98
  6. 2016Understanding and Improving Convolutional Neural Networks via Concatenated Rectified Linear Unitscited by 86
  7. 2021Continual Backprop: Stochastic Gradient Descent with Persistent Randomnesscited by 80
  8. 2020Reinforcement Learning for Non-Stationary Markov Decision Processes: The Blessing of (More) Optimismcited by 76
  9. 2023Deep Reinforcement Learning with Plasticity Injectioncited by 74
  10. 2022SGD with Large Step Sizes Learns Sparse Featurescited by 68
  11. 2020Lipschitz and Comparator-Norm Adaptivity in Online Learningcited by 57
  12. 2021Impossible Tuning Made Possible: A New Expert Algorithm and Its Applicationscited by 52
  13. 2022Learning threshold neurons via the "edge of stability"cited by 51
  14. 2014Unconstrained Online Linear Learning in Hilbert Spaces: Minimax Algorithms and Normal Approximationscited by 46
  15. 2022Parameter-free Mirror Descentcited by 45
  16. 2017Parameter-free online learning via model selectioncited by 34
  17. 2020Lifelong Policy Gradient Learning of Factored Policies for Faster Training Without Forgettingcited by 29
  18. 2022PDE-Based Optimal Strategy for Unconstrained Online Learningcited by 27
  19. 2023Resetting the Optimizer in Deep RL: An Empirical Studycited by 26
  20. 2020Never Stop Learning: The Effectiveness of Fine-Tuning in Robotic Reinforcement Learningcited by 23
  21. 2019Combining Online Learning Guaranteescited by 22
  22. 2022Lifelong Reinforcement Learning with Modulating Maskscited by 22
  23. 2020Online mirror descent and dual averaging: keeping pace in the dynamic casecited by 21
  24. 2023Mechanic: A Learning Rate Tunercited by 20
  25. 2020Model-Free Non-Stationary RL: Near-Optimal Regret and Applications in Multi-Agent RL and Inventory Controlcited by 17
  26. 2024Discounted Adaptive Online Learning: Towards Better Regularizationcited by 12
  27. 2023Sharing Lifelong Reinforcement Learning Knowledge via Modulating Maskscited by 10
  28. 2023Improving Adaptive Online Learning Using Refined Discretizationcited by 8
  29. 2019Multi-Point Bandit Algorithms for Nonstationary Online Nonconvex Optimizationcited by 2
  30. 2021Parameter-free Gradient Temporal Difference Learningcited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 2; 2025: 6; 2026: 9.

17 papers.

Sort:

By area:

  1. 2025Stable Gradients for Stable Learning at Scale in Deep Reinforcement Learningcited by 23
  2. 2025Mitigating Plasticity Loss in Continual Reinforcement Learning by Reducing Churncited by 22
  3. 2025A Survey of Continual Reinforcement Learningcited by 18
  4. 2024Control-oriented Clustering of Visual Latent Representationcited by 8
  5. 2026Agent-Dice: Disentangling Knowledge Updates via Geometric Consensus for Agent Continual Learningcited by 6
  6. 2025Continual Knowledge Adaptation for Reinforcement Learningcited by 5
  7. 2026Non-Uniform Noise-to-Signal Ratio in the REINFORCE Policy-Gradient Estimatorcited by 4
  8. 2026Continual Policy Consolidation for Lifelong Robot Learningcited by 3
  9. 2025Plasticine: Accelerating Research in Plasticity-Motivated Deep Reinforcement Learningcited by 2
  10. 2026Finding the Time to Think: Learning Planning Budgets in Real-Time RLcited by 2
  11. 2024Permutation Invariant Learning with High-Dimensional Particle Filterscited by 1
  12. 2025Rethinking the Role of Dynamic Sparse Training for Scalable Deep Reinforcement Learningcited by 1
5 more
  1. 2026AdamO: A Collapse-Suppressed Optimizer for Offline RLcited by 1
  2. 2026Principled Fast and Meta Knowledge Learners for Continual Reinforcement Learningcited by 0
  3. 2026Vision Transformers that Never Stop Learningcited by 0
  4. 2026An Analysis of Streaming Deep Reinforcement Learning for Adaptive Continual Learning in Roboticscited by 0
  5. 2026Continual Reinforcement Learning with Neuroevolutioncited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Statistical Guarantees for Lifelong Reinforcement Learning using PAC-Bayes Theory
  2. 2026Beyond Single-Model Optimization: Preserving Plasticity in Continual Reinforcement Learning
  3. 2025Instance-Dependent Continuous-Time Reinforcement Learning via Maximum Likelihood Estimation
  4. 2026Benchmarking Reinforcement Learning via Stochastic Converse Optimality: Generating Systems with Known Optimal Policies
  5. 2023TD-MPC2: Scalable, Robust World Models for Continuous Control
  6. 2023Reparameterized Policy Learning for Multimodal Trajectory Optimization
  7. 2026Tape: A Cellular Automata Benchmark for Evaluating Rule-Shift Generalization in Reinforcement Learning
  8. 2024A General Control-Theoretic Approach for Reinforcement Learning: Theory and Algorithms