mlsanityData to 14 Oct 2026

LoGRA: Scaling LLM Reinforcement Learning with Low-Rank Gradient Sketches

arXiv 2610.06647 ↗First posted 5 Oct 2026cs.CLcited by 0

Not on the map. Why?

Abstract (arXiv)

Reinforcement learning has greatly advanced the capabilities of large language models, but its memory demands remain a barrier to broader adoption. We introduce LoGRA, an approach to RL post-training that reduces memory by retaining useful learning signals in low-rank gradient sketches. These compact representations support both model updates and efficient policy synchronization. To prevent overly large updates from disrupting learning, we complement gradient compression with predicted-KL step control, which estimates policy changes before applying each update and adjusts its magnitude accordingly. With all techniques combined, LoGRA reduces average training memory usage by up to 45.7% across reasoning tasks without compromising performance. It also enables stable training of a 27B-parameter model for over 1,100 steps on a single eight-GPU node, where dense Adam runs out of memory, making previously memory-infeasible RL training practical. Code is available in the hrefhttps://github.com/skzhang1/labs-molt/tree/logra/examples/scripts/lograMolt library.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 16 of its 26 references that are in the map. First posted on arXiv 5 Oct 2026.

Builds on

16 of 26 references.

References · 16

  1. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  2. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  3. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  4. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 6,330
  5. 2023Let's Verify Step by Stepcited by 3,789
  6. 2023Qwen Technical Reportcited by 3,344
  7. 2024HybridFlow: A Flexible and Efficient RLHF Frameworkcited by 2,057
  8. 2016Training Deep Nets with Sublinear Memory Costcited by 848
  9. 2023PyTorch FSDP: Experiences on Scaling Fully Sharded Data Parallelcited by 660
  10. 2024GaLore: Memory-Efficient LLM Training by Gradient Low-Rank Projectioncited by 463
  11. 2024Flora: Low-Rank Adapters Are Secretly Gradient Compressorscited by 108
  12. 2024Offline Training of Language Model Agents with Functions as Learnable Weightscited by 48
4 more
  1. 2026OSWorld 2.0: Benchmarking Computer Use Agents on Long-Horizon Real-World Taskscited by 40
  2. 2026ProRL Agent: Rollout-as-a-Service for RL Training of Multi-Turn LLM Agentscited by 20
  3. 2025GaLore 2: Large-Scale LLM Pre-Training by Gradient Low-Rank Projectioncited by 14
  4. 2026Polar: Agentic RL on Any Harness at Scalecited by 14

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025The Art of Scaling Reinforcement Learning Compute for LLMs
  2. 2025Stabilizing Reinforcement Learning with LLMs: Formulation and Practices
  3. 2025Laminar: A Scalable Asynchronous RL Post-Training Framework
  4. 2025RAPID: An Efficient Reinforcement Learning Algorithm for Small Language Models
  5. 2026Sparse-RL: Breaking the Memory Wall in LLM Reinforcement Learning via Stable Sparse Rollouts
  6. 2026GAC: Stabilizing Asynchronous RL Training for LLMs via Gradient Alignment Control
  7. 2026IsoCompute Playbook: Optimally Scaling Sampling Compute for LLM RL
  8. 2026RL Excursions during Pre-Training: Re-examining Policy Optimization for LLM training