# HiLoRe: What to Store, Compress, or Recompute for Efficient GRPO Training

arXiv: https://arxiv.org/abs/2609.33570 · HTML: https://mlsanity.com/paper/2609.33570/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 25 of its 56 references that are in the map.
- First posted on arXiv 27 Sep 2026.

## Placed in

- LLM reasoning

## Builds on (most cited first, 25 of 25)

- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [Training Verifiers to Solve Math Word Problems](https://mlsanity.com/paper/2110.14168/) (2021)
- [DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models](https://mlsanity.com/paper/2402.03300/) (2024)
- [Qwen2.5 Technical Report](https://mlsanity.com/paper/2412.15115/) (2024)
- [Let's Verify Step by Step](https://mlsanity.com/paper/2305.20050/) (2023)
- [Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone](https://mlsanity.com/paper/2404.14219/) (2024)
- [Training Deep Nets with Sublinear Memory Cost](https://mlsanity.com/paper/1604.06174/) (2016)
- [DeepMath-103K: A Large-Scale, Challenging, Decontaminated, and Verifiable Mathematical Dataset for Advancing Reasoning](https://mlsanity.com/paper/2504.11456/) (2025)
- [Logic-RL: Unleashing LLM Reasoning with Rule-Based Reinforcement Learning](https://mlsanity.com/paper/2502.14768/) (2025)
- [TACO: Topics in Algorithmic COde generation dataset](https://mlsanity.com/paper/2312.14852/) (2023)
- [OpenRLHF: An Easy-to-use, Scalable and High-performance RLHF Framework](https://mlsanity.com/paper/2405.11143/) (2024)
- [FP8-RL: A Practical and Stable Low-Precision Stack for LLM Reinforcement Learning](https://mlsanity.com/paper/2601.18150/) (2026)
- [Infinite Sampling: Efficient and Stable Grouped RL Training for Large Language Models](https://mlsanity.com/paper/2506.22950/) (2025)
- [Sparrow: Sparse Rollout for Stable and Efficient Long-context RL of Large Language Models](https://mlsanity.com/paper/2606.08446/) (2026)
- [Adacc: An Adaptive Framework Unifying Compression and Activation Recomputation for LLM Training](https://mlsanity.com/paper/2508.00806/) (2025)
- [Not all tokens are needed(NAT): token efficient reinforcement learning](https://mlsanity.com/paper/2603.06619/) (2026)
- [LongStraw: Long-Context RL Beyond 2M Tokens under a Fixed GPU Budget](https://mlsanity.com/paper/2607.14952/) (2026)
- [QUADS: Stabilizing NVFP4 Reinforcement Learning for MoE via QUantization-error Alignment across Dual Sides](https://mlsanity.com/paper/2607.15810/) (2026)
- [How to Compress KV Cache in RL Post-Training? Shadow Mask Distillation for Memory-Efficient Alignment](https://mlsanity.com/paper/2605.06850/) (2026)
- [DualKV: Shared-Prompt Flash Attention for Efficient RL Training with Large Rollouts and Long Contexts](https://mlsanity.com/paper/2605.15422/) (2026)
- [OASIS: Online Activation Subspace Learning for Memory-Efficient Training](https://mlsanity.com/paper/2604.09406/) (2026)
- [Cost-Aware Learning](https://mlsanity.com/paper/2604.28020/) (2026)
- [Activation Compression in LLMs: Theoretical Analysis and Efficient Algorithm](https://mlsanity.com/paper/2605.01255/) (2026)
- [Schedule-Level Shared-Prefix Reuse for LLM RL Training](https://mlsanity.com/paper/2606.01143/) (2026)
- [LazyTrain: Limited-resource Allocation toward Zero-waste Yield Optimization in Large Language Model Training](https://mlsanity.com/paper/2608.11919/) (2026)

## Abstract (arXiv)

Group-relative policy optimization (GRPO) makes learner-side activations a major memory-computation bottleneck: gradient checkpointing reduces activation memory through recomputation, but fixed schedules can leave roughly 18 GB unused on a 48-GB GPU despite substantial recomputation overhead. Existing activation-management methods set state fidelity from execution cost, tensor properties, or generic compression sensitivity, without explicitly incorporating GRPO's analytic update structure into state-fidelity allocation. We formalize this dependence as policy-update exposure, linking the current GRPO loss coefficients to state-level approximation sensitivity. These coefficients are available before backward without an additional backward pass. We introduce HiLoRe, which allocates graph-attributed recovery units among high-precision storage, low-precision compression, and deterministic recomputation using measured recovery utility and update-conditioned approximation risk. It combines high-precision storage and deterministic recomputation with low-precision recovery under a calibrated risk budget. Across five model-task settings with 2K responses and memory < 1.10 times GC's per-GPU actor-update peak, HiLoRe's actor-update throughput gains reach 13.5% over GC and 7.9% over the fastest evaluated baseline, with paired mean downstream-score differences below 0.6 percentage points.