# FlexLoop: Depth-Elastic Looped Policies for Adaptive Test-Time Computation in Deep RL

arXiv: https://arxiv.org/abs/2609.34488 · HTML: https://mlsanity.com/paper/2609.34488/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning.
- Builds on 20 of its 26 references that are in the map.
- First posted on arXiv 28 Sep 2026.

## Placed in

- LLM reasoning / [Reinforcement learning for reasoning](https://mlsanity.com/lr/reinforcement-learning-for-reasoning/)

## Builds on (most cited first, 20 of 20)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Scaling Laws for Neural Language Models](https://mlsanity.com/paper/2001.08361/) (2020)
- [RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control](https://mlsanity.com/paper/2307.15818/) (2023)
- [OpenVLA: An Open-Source Vision-Language-Action Model](https://mlsanity.com/paper/2406.09246/) (2024)
- [Emergent Abilities of Large Language Models](https://mlsanity.com/paper/2206.07682/) (2022)
- [Offline Reinforcement Learning with Implicit Q-Learning](https://mlsanity.com/paper/2110.06169/) (2021)
- [Universal Transformers](https://mlsanity.com/paper/1807.03819/) (2018)
- [Scaling Latent Reasoning via Looped Language Models](https://mlsanity.com/paper/2510.25741/) (2025)
- [Value Iteration Networks](https://mlsanity.com/paper/1602.02867/) (2016)
- [Looped Transformers are Better at Learning Learning Algorithms](https://mlsanity.com/paper/2311.12424/) (2023)
- [The Predictron: End-To-End Learning and Planning](https://mlsanity.com/paper/1612.08810/) (2016)
- [LoopFormer: Elastic-Depth Looped Transformers for Latent Reasoning via Shortcut Modulation](https://mlsanity.com/paper/2602.11451/) (2026)
- [1000 Layer Networks for Self-Supervised RL: Scaling Depth Can Enable New Goal-Reaching Capabilities](https://mlsanity.com/paper/2503.14858/) (2025)
- [Latent Thinking Optimization: Your Latent Reasoning Language Model Secretly Encodes Reward Signals in Its Latent Thoughts](https://mlsanity.com/paper/2509.26314/) (2025)
- [ELT: Elastic Looped Transformers for Visual Generation](https://mlsanity.com/paper/2604.09168/) (2026)
- [TreeQN and ATreeC: Differentiable Tree-Structured Models for Deep Reinforcement Learning](https://mlsanity.com/paper/1710.11417/) (2017)
- [Adaptive Depth in Looped Transformers: Diagnosing Learned Halting Gates and Trajectory Readouts](https://mlsanity.com/paper/2607.20519/) (2026)
- [Depth-adaptive Inference of Looped Language Models via Continuous Depth Batching](https://mlsanity.com/paper/2608.09444/) (2026)
- [Is Temporal Difference Learning the Gold Standard for Stitching in RL?](https://mlsanity.com/paper/2510.21995/) (2025)
- [On the Role of Computation in Reinforcement Learning](https://mlsanity.com/paper/2602.05999/) (2026)

## Abstract (arXiv)

Looped architectures scale computation by reusing the same parameters across recurrent steps, and recent work shows that they substantially improve deep reinforcement learning policies on long-horizon tasks. Since recurrent depth directly controls computation, one may expect looped policies to naturally support elastic inference across recurrent depths. Surprisingly, we find that pretrained looped policies exhibit severe recurrent-depth specialization: reliable decisions are concentrated near the full trained depth, tying deployment computation to this depth even when less computation may suffice. Achieving depth elasticity, i.e., reliable decisions across recurrent depths with adaptive computation at deployment, therefore remains a key challenge. To address this, we propose FlexLoop, a novel post-training framework that converts pretrained fixed-depth looped policies into depth-elastic policies. FlexLoop keeps training on the original RL objective to preserve full-depth capability while performing adjacent-depth policy distillation to progressively transfer decision quality from deeper to shallower recurrent steps. The resulting policy supports reliable inference across recurrent depths and enables state-wise adaptive inference through recurrent-depth consistency. Experiments on 30 online and offline long-horizon goal-conditioned environments show that FlexLoop preserves full-depth performance while making shallower depths effective. Keeping competitive performance, FlexLoop reduces average recurrent depth by up to bf43% and achieves up to bf1.34× wall-clock speedup in a stress test.