mlsanityData to 14 Oct 2026

Memory-Efficient Policy Libraries with Low-Rank Adaptation in Reinforcement Learning

arXiv 2606.25700 ↗First posted 24 Jun 2026cs.LG cs.ROcited by 0

On the map: Deep reinforcement learning methods → Deep RL algorithms and representations

Abstract (arXiv)

When fine-tuning Large Language Models (LLMs), there has been success in minimizing both memory usage and computation with Parameter-Efficient Fine-Tuning (PEFT), like Low Rank Adaptation (LoRA). In this article, we have explored whether this approach is transferable to the world of robotics and Reinforcement Learning (RL), allowing learning with reduced memory usage and improved computational performance. Specifically, we focused on a version of multi-task robotics, where a library of specialist policies are created. In such a library memory efficiency is especially important. We used a Proximal Policy Optimization (PPO) algorithm and fine-tuned a baseline model to different tasks using LoRA. Our results demonstrate that, depending on the hyperparameters, LoRA can minimize memory usage by a factor of 20-160 compared to full fine-tuning of all layers. This implies a 90-95% storage saving when deploying a library of many (10-50) specialized policies, which can be the differentiating factor between being able to store the entire library in memory or having to use swap-memory in an applied robotics setting. At the same time, our results indicate that there is no significant difference in the success-rate between full fine-tuning and LoRA fine-tuning for the selected tasks.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Deep RL algorithms and representations, one of 15 directions in Deep reinforcement learning methods. Builds on 13 of its 20 references that are in the map. First posted on arXiv 24 Jun 2026.

Builds on

13 of 20 references.

References · 13

  1. 2017Proximal Policy Optimization Algorithmscited by 15,215
  2. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  3. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
  4. 2015Trust Region Policy Optimizationcited by 2,580
  5. 2016Asynchronous Methods for Deep Reinforcement Learningcited by 2,258
  6. 2020Gradient Surgery for Multi-Task Learningcited by 1,256
  7. 2019Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learningcited by 1,204
  8. 2023A Comprehensive Survey of Continual Learning: Theory, Method and Applicationcited by 872
  9. 2017Overcoming Catastrophic Forgetting by Incremental Moment Matchingcited by 243
  10. 2016Learning Modular Neural Network Policies for Multi-Task and Multi-Robot Transfercited by 127
  11. 2025A Comprehensive Survey of Mixture-of-Experts: Algorithms, Theory, and Applicationscited by 91
  12. 2024Multi-Task Reinforcement Learning for Quadrotorscited by 20
1 more
  1. 2025Multi-Task Reinforcement Learning with Language-Encoded Gated Policy Networkscited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2023Efficient RLHF: Reducing the Memory Usage of PPO
  2. 2026Adaptive Capacity Allocation for Vision Language Action Fine-tuning
  3. 2026SLowRL: Safe Low-Rank Adaptation Reinforcement Learning for Locomotion
  4. 2026Low-Rank Adaptation for Critic Learning in Off-Policy Reinforcement Learning
  5. 2026Hybrid-LoRA: Bridging Full Fine-Tuning and Low-Rank Adaptation for Post-Training
  6. 2024Parameter Efficient Reinforcement Learning from Human Feedback
  7. 2026When are LLMs Sufficient Policy Optimizers for Sequential RL Tasks?
  8. 2026Strategic Over-Parameterization for Generalizable Low-Rank Adaptation