mlsanityData to 14 Oct 2026

Low-Rank Adaptation for Critic Learning in Off-Policy Reinforcement Learning

arXiv 2604.18978 ↗First posted 21 Apr 2026cs.LG cs.AIcited by 0

On the map: Deep reinforcement learning methods → Deep RL algorithms and representations

Abstract (arXiv)

Scaling critic capacity is a promising direction for improving off-policy reinforcement learning (RL). However, recent work shows that larger critics are prone to overfitting and instability in replay-based bootstrapped training. In this paper, we propose using Low-Rank Adaptation (LoRA) as a structural regularizer for critic learning. Our approach freezes randomly initialized base matrices and optimizes only the corresponding low-rank adapters, thereby constraining critic updates to a low-dimensional subspace. We evaluate our method across different off-policy RL algorithms, including SAC and FastTD3 based on different network architectures. Empirically, LoRA efficiently reduces critic loss during training and improves overall policy performance, achieving the best or competitive results on most tasks. Extensive experiments demonstrate that our low-rank updates provide a simple and effective form of structural regularization for critic learning in off-policy RL.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Deep RL algorithms and representations, one of 15 directions in Deep reinforcement learning methods. Builds on 28 of its 31 references that are in the map. First posted on arXiv 21 Apr 2026.

Builds on

28 of 31 references.

Closest ancestors · 6

Same direction, newest first

  1. 2025Network Sparsity Unlocks the Scaling Potential of Deep Reinforcement Learning20 Jun
  2. 2025Hyperspherical Normalization for Scalable Deep Reinforcement Learning21 Feb
  3. 2025Scaling Off-Policy Reinforcement Learning with Batch and Weight Normalization11 Feb
  4. 2024SimBa: Simplicity Bias for Scaling Up Parameters in Deep Reinforcement Learning13 Oct
  5. 2024Bigger, Regularized, Optimistic: scaling for compute and sample-efficient continuous control25 May
  6. 2024Stop Regressing: Training Value Functions via Classification for Scalable Deep RL6 Mar

Other references · 22

  1. 2017Proximal Policy Optimization Algorithmscited by 15,215
  2. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  3. 2020Scaling Laws for Neural Language Modelscited by 6,586
  4. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
  5. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,417
  6. 2015Deep Reinforcement Learning with Double Q-learningcited by 1,745
  7. 2021Isaac Gym: High Performance GPU-Based Physics Simulation For Robot Learningcited by 998
  8. 2018DeepMind Control Suitecited by 956
  9. 2017A Distributional Perspective on Reinforcement Learningcited by 765
  10. 2023TD-MPC2: Scalable, Robust World Models for Continuous Controlcited by 512
  11. 2023Orbit: A Unified Simulation Framework for Interactive Robot Learning Environmentscited by 295
  12. 2022The Primacy Bias in Deep Reinforcement Learningcited by 247
10 more
  1. 2023Understanding plasticity in neural networkscited by 174
  2. 2020What Matters In On-Policy Reinforcement Learning? A Large-Scale Empirical Studycited by 162
  3. 2022On the Effectiveness of Parameter-Efficient Fine-Tuningcited by 141
  4. 2020Implicit Under-Parameterization Inhibits Data-Efficient Deep Reinforcement Learningcited by 134
  5. 2021Dropout Q-Functions for Doubly Efficient Reinforcement Learningcited by 114
  6. 2019CrossQ: Batch Normalization in Deep Reinforcement Learning for Greater Sample Efficiency and Simplicitycited by 95
  7. 2024nGPT: Normalized Transformer with Representation Learning on the Hyperspherecited by 70
  8. 2025FastTD3: Simple, Fast, and Capable Reinforcement Learning for Humanoid Controlcited by 44
  9. 2025Bigger, Regularized, Categorical: High-Capacity Value Functions are Efficient Multi-Task Learnerscited by 23
  10. 2026RN-D: Discretized Categorical Actors for On-Policy Reinforcement Learningcited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Normalized Low-Rank Adaptation
  2. 2021Offline RL Without Off-Policy Evaluation
  3. 2024Model-free Low-Rank Reinforcement Learning via Leveraged Entry-wise Matrix Estimation
  4. 2021Offline Reinforcement Learning with Fisher Divergence Critic Regularization
  5. 2024Optimistic Critic Reconstruction and Constrained Fine-Tuning for General Offline-to-Online RL
  6. 2020Critic Regularized Regression
  7. 2024Overestimation, Overfitting, and Plasticity in Actor-Critic: the Bitter Lesson of Reinforcement Learning
  8. 2023A Connection between One-Step Regularization and Critic Regularization in Reinforcement Learning