# Skip a Layer or Loop it? Test-Time Depth Adaptation of Pretrained LLMs

arXiv: https://arxiv.org/abs/2507.07996 · HTML: https://mlsanity.com/paper/2507.07996/

## Key facts

- Cited by 17 papers in the map (data through 14 Oct 2026).
- Builds on 21 of its 26 references that are in the map.
- First posted on arXiv 10 Jul 2025.

## Builds on (most cited first, 21 of 21)

- [Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge](https://mlsanity.com/paper/1803.05457/) (2018)
- [Scaling Language Models: Methods, Analysis & Insights from Training Gopher](https://mlsanity.com/paper/2112.11446/) (2021)
- [BranchyNet: Fast Inference via Early Exiting from Deep Neural Networks](https://mlsanity.com/paper/1709.01686/) (2017)
- [Faith and Fate: Limits of Transformers on Compositionality](https://mlsanity.com/paper/2305.18654/) (2023)
- [Reducing Transformer Depth on Demand with Structured Dropout](https://mlsanity.com/paper/1909.11556/) (2019)
- [Scaling up Test-Time Compute with Latent Reasoning: A Recurrent Depth Approach](https://mlsanity.com/paper/2502.05171/) (2025)
- [Neural Module Networks](https://mlsanity.com/paper/1511.02799/) (2015)
- [DeeBERT: Dynamic Early Exiting for Accelerating BERT Inference](https://mlsanity.com/paper/2004.12993/) (2020)
- [BERT Loses Patience: Fast and Robust Inference with Early Exit](https://mlsanity.com/paper/2006.04152/) (2020)
- [FastBERT: a Self-distilling BERT with Adaptive Inference Time](https://mlsanity.com/paper/2004.02178/) (2020)
- [Compressing BERT: Studying the Effects of Weight Pruning on Transfer Learning](https://mlsanity.com/paper/2002.08307/) (2020)
- [Looped Transformers are Better at Learning Learning Algorithms](https://mlsanity.com/paper/2311.12424/) (2023)
- [Looped Transformers for Length Generalization](https://mlsanity.com/paper/2409.15647/) (2024)
- [DART-Math: Difficulty-Aware Rejection Tuning for Mathematical Problem-Solving](https://mlsanity.com/paper/2407.13690/) (2024)
- [On Limitations of the Transformer Architecture](https://mlsanity.com/paper/2402.08164/) (2024)
- [Inner Thinking Transformer: Leveraging Dynamic Depth Scaling to Foster Adaptive Internal Thinking](https://mlsanity.com/paper/2502.13842/) (2025)
- [You Need Multiple Exiting: Dynamic Early Exiting for Accelerating Unified Vision Language Model](https://mlsanity.com/paper/2211.11152/) (2022)
- [LGViT: Dynamic Early Exiting for Accelerating Vision Transformer](https://mlsanity.com/paper/2308.00255/) (2023)
- [Mixture of Nested Experts: Adaptive Processing of Visual Tokens](https://mlsanity.com/paper/2407.19985/) (2024)
- [Anytime Dense Prediction with Confidence Adaptivity](https://mlsanity.com/paper/2104.00749/) (2021)
- [DACT-BERT: Differentiable Adaptive Computation Time for an Efficient BERT Inference](https://mlsanity.com/paper/2109.11745/) (2021)

## Built on by (most cited first, 17 of 17)

- [Memory-Efficient Looped Transformer: Decoupling Compute from Memory in Looped Language Models](https://mlsanity.com/paper/2605.07721/) (2026)
- [Dr.LLM: Dynamic Layer Routing in LLMs](https://mlsanity.com/paper/2510.12773/) (2025)
- [LoopRPT: Reinforcement Pre-Training for Looped Language Models](https://mlsanity.com/paper/2603.19714/) (2026)
- [Skip a Layer or Loop It? Learning Program-of-Layers in LLMs](https://mlsanity.com/paper/2606.06574/) (2026)
- [Allocating Recurrent Compute in Looped Language Models](https://mlsanity.com/paper/2608.18230/) (2026)
- [The Topological Trouble With Transformers](https://mlsanity.com/paper/2604.17121/) (2026)
- [Catch Your Breath: Adaptive Computation for Self-Paced Sequence Production](https://mlsanity.com/paper/2510.13879/) (2025)
- [MACRO: Markov Chain Routing of Transformer Layers](https://mlsanity.com/paper/2608.05872/) (2026)
- [Towards Understanding Layer Contributions in Tabular In-Context Learning Models](https://mlsanity.com/paper/2511.15432/) (2025)
- [Programs-of-Layers in LLMs through the Lens of Cortical Areas](https://mlsanity.com/paper/2609.31360/) (2026)
- [Architectural Sampling: Test-Time Scaling via Computational Diversity in Frozen Vision-Language Models](https://mlsanity.com/paper/2610.01687/) (2026)
- [Ladder Up, Memory Down: Low-Cost Fine-Tuning With Side Nets](https://mlsanity.com/paper/2512.14237/) (2025)
- [When LLM Meets Tree Search: A Systematic View of Inference as Search in Large Language Models](https://mlsanity.com/paper/2608.30395/) (2026)
- [Write Back the Delta: Revisiting the Same Tokens with Fresh Representations](https://mlsanity.com/paper/2609.32457/) (2026)
- [Random Recursive Models](https://mlsanity.com/paper/2610.00541/) (2026)
- [Dynamic Routing as a New Dimension for Test-time Versatility of LLMs](https://mlsanity.com/paper/2610.04751/) (2026)
- [Input-Blind Controls Produce Substantial Oracle Headroom for Layer Programs in Multiple-Choice Evaluation](https://mlsanity.com/paper/2610.10368/) (2026)

## Abstract (arXiv)

Can a pretrained neural network adapt its architecture to different inputs without any finetuning? Do we need all layers for simple tasks, and are they adequate for challenging tasks? We found that the layers of a pretrained large language model (LLM) can be manipulated as separate modules to build a better and even shallower model customized for each test sample. In particular, each layer from the pretrained model can be skipped/pruned or repeated multiple times as recurrent neural networks (RNN), and stacked with others in arbitrary orders, yielding a chain-of-layers (CoLa) per sample. This compositional space greatly expands the scope of existing works on looped/recurrent pretrained modules, layer pruning, or early-exit networks. We develop a Monte Carlo Tree Search (MCTS) protocol to explore and identify the optimal CoLa for each sample from math and commonsense reasoning benchmarks. Compared to a static model of a fixed depth, CoLa allows shortcut paths (fast thinking), recurrence of the same layer(s) (slow thinking), and combining both, offering more flexible, dynamic architectures for different inputs. We conduct an extensive analysis of the MCTS-optimized CoLa, which leads to two key findings: (1) For >75% of samples with correct predictions by the original LLM, we can find shorter CoLa, suggesting a large space for improving inference efficiency; (2) For >60% of samples with originally incorrect predictions, we can identify CoLa achieving correct predictions, suggesting a large space of performance enhancement. Our results highlight the shortcomings of using a fixed architecture of pre-trained LLMs for inference on different samples and pave the way to unlock the generalization power of test-time depth adaptation.