mlsanityData to 7 Oct 2026

AdamO: A Collapse-Suppressed Optimizer for Offline RL

arXiv 2605.01968 ↗First posted 3 May 2026cs.LGcited by 1

On the map: Offline RL and imitation learning from data → Offline RL algorithms

Abstract (arXiv)

Offline reinforcement learning (RL) can fail spectacularly when bootstrapped temporal-difference (TD) updates amplify their own errors, driving the critic toward extreme and unusable Q-values. A key counterintuitive insight of this work is that collapse is not only a property of the backup rule or network architecture: optimizer dynamics themselves can directly trigger or suppress instability. From a control-theoretic viewpoint, we model offline TD learning as a feedback system and analyze Adam-based critic updates. This yields a necessary and sufficient condition for stability of the induced local update dynamics: within the regime we analyze, these dynamics are stable if and only if the spectral radius of the corresponding update operator is strictly below one. Further analysis suggests that standard Adam updates can inadvertently distort the parameter geometry, motivating explicit orthogonality constraints to prevent TD error amplification. To this end, we propose AdamO, an Adam-based optimizer with a decoupled orthogonality correction regulated by a strict task-alignment budget. We prove that this design theoretically guarantees worst-case task safety and preserves Adam's continuous-time dissipative dynamics. Empirically, AdamO is broadly compatible with diverse offline RL baselines, improving stability and returns across a broad suite of benchmarks.

Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Placed in Offline RL algorithms, one of 13 directions in Offline RL and imitation learning from data. Builds on 39 of its 55 references that are in the map. First posted on arXiv 3 May 2026.

Builds on

39 of 55 references.

Closest ancestors · 15

Same direction, newest first

  1. 2025FOVA: Offline Federated Reinforcement Learning with Mixed-Quality Data2 Dec
  2. 2025Offline RL with Smooth OOD Generalization in Convex Hull and its Neighborhood10 Jun
  3. 2023Understanding, Predicting and Better Resolving Q-Value Divergence in Offline-RL6 Oct
  4. 2023Revisiting the Minimalist Approach to Offline Reinforcement Learning16 May
  5. 2022Offline Q-Learning on Diverse Multi-Task Data Both Scales And Generalizes28 Nov
  6. 2022Q-Ensemble for Offline RL: Don't Scale the Ensemble, Scale the Batch Size20 Nov
  7. 2022CORL: Research-oriented Deep Offline Reinforcement Learning Library13 Oct
  8. 2022Mildly Conservative Q-Learning for Offline Reinforcement Learning9 Jun
  9. 2021DR3: Value-Based Deep Reinforcement Learning Requires Explicit Regularization9 Dec
  10. 2021Offline Reinforcement Learning with Implicit Q-Learning12 Oct
  11. 2021Uncertainty-Based Offline Reinforcement Learning with Diversified Q-Ensemble4 Oct
  12. 2021A Minimalist Approach to Offline Reinforcement Learning12 Jun
3 more
  1. 2020Conservative Q-Learning for Offline Reinforcement Learning8 Jun
  2. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learning15 Apr
  3. 2019Behavior Regularized Offline Reinforcement Learning26 Nov

Other references · 24

  1. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  2. 2015Continuous control with deep reinforcement learningcited by 3,279
  3. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,597
  4. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,302
  5. 2018Spectral Normalization for Generative Adversarial Networkscited by 1,618
  6. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,227
  7. 2019On the Convergence of Adam and Beyondcited by 834
  8. 2019Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reductioncited by 833
  9. 2024GaLore: Memory-Efficient LLM Training by Gradient Low-Rank Projectioncited by 447
  10. 2023Efficient Online Reinforcement Learning with Offline Datacited by 332
  11. 2022Offline Reinforcement Learning via High-Fidelity Generative Behavior Modelingcited by 177
  12. 2018On the Convergence of A Class of Adam-Type Algorithms for Non-Convex Optimizationcited by 173
12 more
  1. 2023The Dormant Neuron Phenomenon in Deep Reinforcement Learningcited by 171
  2. 2018Deep Reinforcement Learning and the Deadly Triadcited by 170
  3. 2017Scalable trust-region method for deep reinforcement learning using Kronecker-factored approximationcited by 119
  4. 2015Increasing the Action Gap: New Operators for Reinforcement Learningcited by 49
  5. 2024Cautious Optimizers: Improving Training with One Line of Codecited by 42
  6. 2023Resetting the Optimizer in Deep RL: An Empirical Studycited by 26
  7. 2024Compressible Dynamics in Deep Overparameterized Low-Rank Learning & Adaptationcited by 26
  8. 2024From Low Rank Gradient Subspace Stabilization to Low-Rank Weights: Observations, Theories, and Applicationscited by 24
  9. 2025Stable Gradients for Stable Learning at Scale in Deep Reinforcement Learningcited by 22
  10. 2024Fast TRAC: A Parameter-Free Optimizer for Lifelong Reinforcement Learningcited by 16
  11. 2024Can Learned Optimization Make Reinforcement Learning Less Difficult?cited by 14
  12. 2025Penalizing Infeasible Actions and Reward Scaling in Reinforcement Learning with Offline Datacited by 7

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026T5: Twin-Critic Training for Token-Level Thoughts in Reinforcement Mid-Trainingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026COOPO: Cyclic Offline-Online Policy Optimization Algorithm
  2. 2024Optimistic Critic Reconstruction and Constrained Fine-Tuning for General Offline-to-Online RL
  3. 2025A Clean Slate for Offline Reinforcement Learning
  4. 2021Instabilities of Offline RL with Pre-Trained Neural Representation
  5. 2023Towards Robust Offline-to-Online Reinforcement Learning via Uncertainty and Smoothness
  6. 2026Policy Plasticity Matters in Offline-to-Online Reinforcement Learning: Refitting Offline Policies for Online Adaptation
  7. 2024Offline-Boosted Actor-Critic: Adaptively Blending Optimal Historical Behaviors in Deep Off-Policy RL
  8. 2025Belief-Based Offline Reinforcement Learning for Delay-Robust Policy Optimization