mlsanityData to 14 Oct 2026

FlexLoop: Depth-Elastic Looped Policies for Adaptive Test-Time Computation in Deep RL

arXiv 2609.34488 ↗First posted 28 Sep 2026cs.LG cs.AIcited by 0

On the map: Deep reinforcement learning methods → RL training of language and agent models

Abstract (arXiv)

Looped architectures scale computation by reusing the same parameters across recurrent steps, and recent work shows that they substantially improve deep reinforcement learning policies on long-horizon tasks. Since recurrent depth directly controls computation, one may expect looped policies to naturally support elastic inference across recurrent depths. Surprisingly, we find that pretrained looped policies exhibit severe recurrent-depth specialization: reliable decisions are concentrated near the full trained depth, tying deployment computation to this depth even when less computation may suffice. Achieving depth elasticity, i.e., reliable decisions across recurrent depths with adaptive computation at deployment, therefore remains a key challenge. To address this, we propose FlexLoop, a novel post-training framework that converts pretrained fixed-depth looped policies into depth-elastic policies. FlexLoop keeps training on the original RL objective to preserve full-depth capability while performing adjacent-depth policy distillation to progressively transfer decision quality from deeper to shallower recurrent steps. The resulting policy supports reliable inference across recurrent depths and enables state-wise adaptive inference through recurrent-depth consistency. Experiments on 30 online and offline long-horizon goal-conditioned environments show that FlexLoop preserves full-depth performance while making shallower depths effective. Keeping competitive performance, FlexLoop reduces average recurrent depth by up to bf43% and achieves up to bf1.34× wall-clock speedup in a stress test.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in RL training of language and agent models, one of 15 directions in Deep reinforcement learning methods. Builds on 20 of its 26 references that are in the map. First posted on arXiv 28 Sep 2026.

Builds on

20 of 26 references.

References · 20

  1. 2017Proximal Policy Optimization Algorithmscited by 15,215
  2. 2020Scaling Laws for Neural Language Modelscited by 6,586
  3. 2023RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Controlcited by 3,080
  4. 2024OpenVLA: An Open-Source Vision-Language-Action Modelcited by 2,887
  5. 2022Emergent Abilities of Large Language Modelscited by 2,512
  6. 2021Offline Reinforcement Learning with Implicit Q-Learningcited by 1,293
  7. 2018Universal Transformerscited by 647
  8. 2025Scaling Latent Reasoning via Looped Language Modelscited by 165
  9. 2016Value Iteration Networkscited by 160
  10. 2023Looped Transformers are Better at Learning Learning Algorithmscited by 149
  11. 2016The Predictron: End-To-End Learning and Planningcited by 93
  12. 2026LoopFormer: Elastic-Depth Looped Transformers for Latent Reasoning via Shortcut Modulationcited by 61
8 more
  1. 20251000 Layer Networks for Self-Supervised RL: Scaling Depth Can Enable New Goal-Reaching Capabilitiescited by 45
  2. 2025Latent Thinking Optimization: Your Latent Reasoning Language Model Secretly Encodes Reward Signals in Its Latent Thoughtscited by 19
  3. 2026ELT: Elastic Looped Transformers for Visual Generationcited by 14
  4. 2017TreeQN and ATreeC: Differentiable Tree-Structured Models for Deep Reinforcement Learningcited by 12
  5. 2026Adaptive Depth in Looped Transformers: Diagnosing Learned Halting Gates and Trajectory Readoutscited by 12
  6. 2026Depth-adaptive Inference of Looped Language Models via Continuous Depth Batchingcited by 6
  7. 2025Is Temporal Difference Learning the Gold Standard for Stitching in RL?cited by 4
  8. 2026On the Role of Computation in Reinforcement Learningcited by 3

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026ELASTIC: Efficiently Learning to Adaptively Scale Test-Time Compute for Generative Control Policies
  2. 2026Depth-Entropy Guided Sampling for Training-Free LLM Reasoning
  3. 2025RLoop: An Self-Improving Framework for Reinforcement Learning with Iterative Policy Initialization
  4. 2026Looped Actor: Depth-Recurrent Reasoning Models for Reinforcement Learning
  5. 2026WarpSAC: Towards the Pinnacle of Scalable Off-policy RL by Rethinking Exploration and Exploitation
  6. 2026Does Scaling Reinforcement Learning Really Require More Training?
  7. 2026DEFlow: Deep Equilibrium Flow Maximum-Entropy Reinforcement Learning
  8. 2026SmoothRL: Online Reinforcement Learning During Asynchronous Execution